↧
Statistics collection recommendations – Teradata 14.0 - comment by carrie
↧
Statistics Collection Recommendations – Teradata 14.10, 15.0 and 15.10 - comment by SANJI
Is there a provision to recollect stats based on their names ?
COLLECT STATS <Statistics Name> ON TABLENAME;
↧
↧
Statistics Collection Recommendations – Teradata 14.10, 15.0 and 15.10 - comment by carrie
Hi Sanji,
Yes, you can recollect statistics by name, if a name exists. This is particularly useful if you have multicolumn statistics.
From page 7 of the orange book titled: Teradata Database 14.10 Statistics Enhancements:
"When you collect statistics on an index or column set, specifying a name for them is optional. A name is mandatory if statistics are for anything other than column references on expressions.
You can later use the name for recollections, copies, transfers, help information, show information, and for dropping the collected statistics. The rules for naming statistics are the same as the rules for naming database objects."
Thanks, -Carrie
↧
Special Treatment For Users DBC and TDWM - comment by geethareddy
Thank you for the advice.
↧
Things that Run in the Session Workload - blog entry by carrie
Cover Image:
You’ll probably want to skip over this posting.
Unless you’re curious about some of the more obscure workload classification conventions used by TASM and TIWM.
In a previous blog posting titled “A New, Simplified Approach to Parsing Workload Selection” I explained how a session workload is determined at the time a session logs onto the database. Once established, the session workload is used to support the activities required for either logging on a session or for parsing queries within the session.
If you have set up a parsing-only workload to serve as a single, high priority session workload for all of your users, and you happen to monitor that workload, you may find things that are not part of session logons or query parsing that are running in that workload.
This blog explains what these other activities are that may show up in a session workload, and some of the oddities in how the session workload is used by workload management routines.
Session Workload
First, let’s be clear on what a session workload does: The session workload manages the access to resources and the priority of activities that take place on the parsing engine, such as logging on a session or parsing a request.
A session workload that was responsible for parsing a query is reported in a field in the DBQLogTbl named SessionWDID. SessionWDID may be the same workload as the WDID for that request. If those two fields are the same, that means that parsing and optimizing for the query took place in the same workload as the AMP execution. Sometimes SessionWDID and WDID are the same workload, sometimes they are not. Read the blog posting mentioned above for more background, and for detail on the session workload and how it is selected.
Now let’s consider some of the activities that you may observe running in SessionWDID.
Stored Procedure CALL Statement
Stored procedure CALL statements always run in the SessionWDID workload for the session. When the dispatcher sees a CALL statement it will ensure it gets put in its SessionWDID so the SessionWDID and the WDID will always be the same for the CALL.
Here’s how a stored procedure CALL would look in DBQLogTbl:
.jpg)
The stored procedure CALL statement incorporates the code and logic, if any, contained in the stored procedure. It is usually a very, very small amount of CPU usage that comes before, between, and after the SQL statements within the stored procedure. A CALL stored procedure is given its own row in the DBQLogTbl, apart from the SQL contained within it.
Below is a different example of DBQL output that illustrates a CALL statement and the three SQL statements within the stored procedure. Note that DBQL in this example reports zero AMP CPU Time for the CALL, and a slight amount of CPU on the parsing engine. The request number for all of the SQL statements within the stored procedure will be the same as the request number for the CALL. However, each SQL statement that is part of the stored procedure will have a distinct internal request number that simply increments the original request number, as shown below.
In earlier releases stored procedure calls ran in the default workload, called “WD-Default”. Starting in 14.10 that changed, as some Teradata sites were concerned when they saw activity in WD-Default after having been careful to set up workload classification in way to avoid anything failing into the default workload. Consequently, there was a change, and today all CALL statements execute in the SessionWDID for the session.
If you would prefer CALL statements to execute in a workload of your choice, and not in the session workload, you can set up classification criteria on the workload where you want the CALL statements to execute. Use the Target Criteria named "Stored Procedure". You can name one or more stored procedures you want to run in that workload, or you can design it so that all stored procedures run in a single workload by using the wild card (*). But remember, we’re talking about only the CALL statement there, the framework of the stored procedure that encapsulates the SQL statements. All the SQL statements within the stored procedure will run in whatever workload they classify to, independent of the CALL statement.
CHECK WORKLOAD Statements
CHECK WORKLOAD statements were introduced in Teradata Database 13.10 to support the load utility unit of work. There are two such statements:
- CHECK WORKLOAD FOR: Precedes the utility unit of work. Passes object information, such as database and table, into the database.
- CHECK WORKLOAD END: Begins the utility unit of work. Determines the number of AMP sessions, triggers workload classification, and performs the utility throttle and resource limit checks.
The CHECK WORKLOAD FOR statement works the same way as the CALL statement in terms of workload classification. CHECK WORKLOAD FOR will always run in SessionWDID (which will be the parsing-only workload if you have one defined). This is very light work, as all that statement does is get the names of the objects involved in the load utility from parser memory, where they are initially read into the database, and pass them to the dispatcher. Similar to a CALL stored procedure, there is no AMP activity involved in the Check Workload For statement.
A CHECK WORKLOAD END statement classifies to the workload where the load utility will execute. It uses a different approach to workload classification than does a CHECK WORKLOAD FOR. A CHECK WORKLOAD END goes through normal request classification and matches to a single workload. Once that workload is selected, the workload becomes the load uiltity’s session workload as well If you look at DBQL output for a load utility, you will see WDID = SessionWDID for the duration of the load.
Here is an example of how CHECK WORKLOAD FOR and CHECK WORKLOAD END look in DBQL:
.jpg)
CHECK WORKLOAD FOR finds SessionWDID and moves it to the WDID, making them the same. CHECK WORKLOAD END does the opposite. It finds the WDID and moves it to the SessionWDID, making them the same for the duration of the load utility.
HELP and SHOW Commands
HELP and SHOW commands are short administrative commands that execute on the parsing engine but may go to the AMPs to get data dictionary information. Typical HELP/SHOW commands include:
- HELP DATABASE
- HELP SESSION
- HELP STATISTICS
- SHOW STATISTICS
- SHOW TABLE
- SHOW VIEW
Most of the time these types of commands will use express requests if they require AMP data. Express requests are streamlined requests that bypass the parser and optimizer and go directly to the data dictionary.
However, if any of these administrative commands are consider large, they will be executed as normal SQL and will be parsed, optimized, and dispatched to the AMPs. This would be the case if a spool file needed to be built to return the data.
If an express request is used to satisfy one of these commands, as is likely the case with a HELP DATABASE or HELP SESSION, then you will see only a SessionWDID in DBQL data. The WDID will be NULL. An express request bypasses the dispatcher, which is the module where the classification to a WDID is performed. So it never undergoes workload classification.
If, however, the administrative command is satisfied by issuing SQL, which might be the case when a lengthy histogram is returned from a SHOW STATISTICS command, it will appear in DBQL with both a SessionWDID and a WDID. They may be the same workload or a different workload.
Here is how DBQL output might look for a series of HELP and SHOW commands:
Notice that some of these commands have a WDID and some do not.
Throttles Can Delay HELP/SHOW Commands
There’s something else to be aware of concerning these administrative commands. Sometimes they end up in the throttle delay queue.
Throttle rules are enforced within the dispatcher before the first step in a request is sent to the AMPs. Because express requests bypass the dispatcher, they are never subject to being delayed by a workload management throttle rule. This means that most HELP/SHOW commands, at least the ones that only display a SessionWDID but not a WDID, are not at risk of being placed in the delay queue.
However, the HELP/SHOW commands that run as regular SQL (the ones that do display a WDID) could be sent to the delay queue. Whether this happens or not depends on whether such commands classify to a system throttle rule, and whether that system throttle is at its limit at the time the SQL reaches the dispatcher. Since a virtual partition throttle is a variation of a system throttle, such an unexpected delay could also happen as a result of a virtual partition throttle.
You can see this use of SQL instead of express requests yourself by running an EXPLAIN on a HELP or a SHOW command:
This HELP command will use SQL and can tentatively be delayed:
EXPLAIN HELP DATABASE cab;
1) First, we lock DBC.TVM for access.
2) Next, we do an all-AMPs RETRIEVE step from DBC.TVM by way of an
all-rows scan with a condition of ("DBC.TVM.Field_1 = '00002F04'XB")
into Spool 1 (group_amps), which is built locally on the AMPs.
Then we do a SORT to order Spool 1 by the sort key in spool field1
(DBC.TVM.NameI).
3) Finally, we send out an END TRANSACTION step to all AMPs involved
in processing the request.
-> The contents of Spool 1 are sent back to the user as the result of statement 1.
This SHOW command uses an express request and will not be delayed:
EXPLAIN SHOW TABLE nation;
-> The row is sent directly back to the user as the result of statement 1.
Preventing Delays
It is simple to prevent these administrative types of commands from ever being delayed. Refine the classification on any system throttles that could be the cause of such a delay by adding critieria that these commands cannot match.
The chart below is a subset of the earlier chart, displaying only the HELP/SHOW commands that have both a SessionWDID and a WDID, an indication that they were converted to SQL:
One thing they have in common is an estimated processing time of zero. Knowing that, you can add classification criteria to any system throttles that could cause these commands to be delayed. Just include this additional specification: “EstProcTime > 0”. That way only requests with an actual estimate will be under the control of the throttle, but HELP and SHOW commands will not.
A future enhancement will automatically bypass throttles for such commands.
Conclusion
A session classifies to a session workload to do the logon processing and also to do the parsing engine work of all requests that execute within that session. However, there are additional types of very light work that will run in the session workload as well.
If you have set up a parsing-only workload, the additional commands and requests that execute at the session priority should not be a performance concern.
Ignore ancestor settings:
0
Apply supersede status to children:
0
↧
↧
A Closer Look at How to Set up a Parsing-Only Workload - comment by carrie
The purpose of the Parsing-only workload is to elevate the priority of CPU usage on the parsing engine when a session logs on or a query undergoes parsing/optimization. This CPU usage on the parsing engine has nothing to do with AMP worker tasks. The PE is a separate module from the AMP. PE-level CPU is consumed by special tasks on the parsing engine, unrelated to AMP worker tasks.
See my blog posting that describes parsing engine tasks: http://developer.teradata.com/blog/carrie/2012/11/database-tasks-that-run-on-the-parsing-engine
Expediting express requests only has an impact to express requests that are sent from the parsing engine to the AMPs when data dictionary access is necessary during parsing-level activities. It could speed of AMP activities that originate from parsing/optimizing, etc. if the system is out of AMP worker tasks on the AMP.
These are two separate capabilties. One or the other may be helpful, or both may be helpful, it depends on your environment and what problem you are trying to solve.
Thanks, -Carrie
↧
Converting CPU Seconds, Old Node Type to New Node Type - comment by gskaushik
Hi Carrie,
Kindly inform on how to calculate the TPERF of a system from RESUSAGE tables.
Thanks in advance.
↧
Converting CPU Seconds, Old Node Type to New Node Type - comment by carrie
Subramanian,
I'm not familiar with the formula or calculations involved in determing the TPERF value for a given hardware node. I believe those are internally determined by Teradata Engineering.
The best way to get the TPERF rating for your nodes is to talk to your Teradata account team or your Customer Support representative.
Thanks, -Carrie
↧
How Resources are Shared in the SLES 11 Priority Scheduler - comment by TrickyTera4u
Hi Carrie,
Nice and detail explanation. Thanks.
Does SLES 11 consume more system CPU? We recently moved to SLES 11 priority scheduler. After the migration, we see good amount of increase in CPUUSERV (system CPU). Earlier it was around 9 to 12% of total CPU and now it is around 20 to 32%. Because of the increase, the CPU utilization of the system is always above 95% even though the execution CPU is well under 65%. I went through one year of history data and I haven't seen more than 15% of system CPU anytime in my system. What could be the possible reason for this? Does it run any additional services in the backend or kernel ? Anyone else experienced the same?
↧
↧
How Resources are Shared in the SLES 11 Priority Scheduler - comment by carrie
Amir,
It is difficult to quantify CPU consumption changes between SLES 10 and SLES 11 because the accounting is completely different in SLES 11.
But in general, most sites experience some increase in CPU usage, much of it in "system". As with all new versions of software that adds improved features and greater value, you will see a slight uptick in CPU in SLES 11 just to account for the extra work being performed. There is, for example, an additional cost to the improved accuracy and granularity in keeping track of resource usage, and with the stonger enforcement of priorities. The new operating system accounting and scheduling is orders of magnitude more precise.
In addition, many of the small things that get done in and by the operating system have been moved to different places in the code in SLES11. SLES11 is a complete re-architecture, and many things that ran somewhere else, now get bucketed into "system". And there are a number of file system background tasks that run to a greater degree in system in SLES 11 compared to SLES 10. So a comparison of that metric before and after is unfortunately apples to oranges.
Thanks, - Carrie
↧
New, Simplified Approach to Parsing Workload Selection - comment by sureshbadam
Carrie,
Thanks for greate articles / blogs on DBQL. I have a situation that is somewaht disscussed above but not matching with my case.
Here is my example
wdid=? sessionwdid=41 and statement=SELECT. I do see delay times in these conditions. By readig above, I understand that if WDID is null and sessionWDID is not null then they are not AMP operations (like HELP, SHOW) but in my situation it is a SELECT. I am trying to fix this issue to avoid delay times.
Any help appreciated..
Thanks
Suresh
↧
New, Simplified Approach to Parsing Workload Selection - comment by carrie
Suresh,
Could this be a PREPARE statement, if it is coming from ODBC.
Based on ODBC settings, queries will get submitted twice. The first time for parsing and second rime for actual execution. You can recognize Prepare queries because they have a WDID as NULL and AMPCPUTime with zero, but will have corresponding ParserCPUTime. Queries which represent the actual execution will have a non-NULL WDID and will show AMPCPUTime.
If the request doesn't go the AMPs (as evidenced by having NULL in WDID), then a throttle could not delay that request. Throttles are enforced at the dispatcher level before the first step in the request is sent to the AMPs, but requests with WDID of NULL never pass through the dispatcher module.
There is more information on this and the possibilites of unexpected delays on the blog posting called: Things That Run In Session Workload.
http://developer.teradata.com/blog/carrie/2015/11/things-that-run-in-the-session-workload
Thanks, -Carrie
↧
Expediting Express Requests - comment by StevenSchmid
Hi Carrie
I'm currently working on a 13.10 system, which is due to be upgraded to 15.0 at the end of the year. We are using Viewpoint 15.0
The system is currently CPU bound and frequently goes into flow control, but one interesting thing I have seen in Workload Designer, is that all the workloads have reserved AWTs = 0, and Max Expedited AWTs = 62.
What would the recommended setting be for "Max Expedited AWTs", and should it only be set for tactical/short type queries ?
Cheers
Steven
↧
↧
Expediting Express Requests - comment by carrie
Steven,
I am assuming you are on SLES 10, as you are on 13.10 database software.
The two settings you mention ("Reserved AWT" and "Max Expedited AWTs") are not set at the workload level. They are set in one place as a sort of global setting, on the Viewpoint Priority Distribution screen. Once they are set, they will be applied to all workloads with the tactical enforcement priority. Non-tactical workloads will not be impacted by those two settings.
So if you don't have any tactical workloads, it doesn't really matter how you set them.
The default for Max Expedited AWTs is 50, the same as the limit of AWTs allowed to support WorkNew work type messages. All the Max Expedited AWTs does is put a limit on the number of AWTs that can be used at any point in time to support new expedited work, which runs in Work08 work type. Requests coming from expedited workloads (workloads with the tactical enforcement priority), will always use worktypes 8, 9 and 10, rather than work types 0, 1, and 2. Even if there are no reserved AWTs specified. This means tasks coming from expedited workloads will be in the work message queue ahead of new work that is not expedited, if there is AWT exhaustion.
This is explained in the most recent AMP Worker Task orange book, Section 4.5, if you can find that in the orange book repository. The title is:
AMP Worker Tasks and ResUsage Monitoring for Teradata 14.0.
I know you are only 13.10 just now, but most of the content of this orange book will apply to you, except some of the ResUsage table fields you won't see until you get to 15.0.
If you do have tactical workloads, you might want to think about using reserved AWTs so that tactical work can always run even if non-tactical work is delayed by flow control episodes. Those tradeoffs are explained in the AWT orange book as well.
In general, the Max Expedited AWTs is fine at the default of 50, but if you expect the concurrency level of new expedited work to exceed 50, and you want to minimize blocking waiting for an AWT for those types of messages, you could increase Max Expedited AWTs to the expected concurrency levels. A better approach would be to analyze ResUsageSawt data and look at the max inuse counts for work08 at your busiest times, and if the counts are less than 50, leave 50 as the default. If the counts are above 50 then you could increase the setting to be a point or two above the max work08 count number.
And if you want to expedite express requests once you get on 15.0, as described in this blog posting, you will need to specify at least 2 in the Reserved AWT setting, even if you do not have tactical work.
Thanks, -Carrie
↧
Expediting Express Requests - comment by StevenSchmid
Hi Carrie
Thanks for your detailed reply.
Yes, we are on SLES10 TD13.10, will upgrade to SLES11 sometime in the new year, after the 15.0 upgrade.
I have found the Orange book you mentioned, very informative.
I have a question, if tactical workloads are not defined, does that mean that the AWTs assigned to Work08,09,10 will go unused, or with 13.10 do tactical queries go to Work01 ? Or, if Work08,09,10 are active, are they only assigned AWTs if the "Reserved AWTs" are assigned a number > 0 ? If Work08,09,10 are assigned AWTs with the "Reserved AWTs=0", will setting up a tactical workload (single/few amp or CPU < 2 secs) make use of the AWTs at no extra AWT overhead, as we currently get flow control. I will do some analysis, and if the number of queries classifying into the tactical workload is large, then I can assign some reserved AWTs which will be taken from the 62 AWT reserve pool (default 80 AWT).
Another question if you don't mind, is I was also considering increasing the AWTs from 80 to say 90, however we only have 96GB 6650H nodes. I did see on one of your blogs, that this would increase memory on the nodes, I think from memory it would be "30 amps/node x 10 newAWTs x 2 MB = 600 MB/node", which doesn't seem excessive. I can also discuss this with Teradata Customer Services, as the Customer really does not want to invest in additional hardware if at all possible
Cheers
Steven
↧
How Resources are Shared in the SLES 11 Priority Scheduler - comment by sghanw
Hi Carrie,
I have one doubt we are on TD 14.10 and SLES11, currently if a query is running on Step 7 which usually takes 30min to complete and if DBA team manually change workload of that query from Batch long ( low priority Timeshare ) to Batch Short ( SLG Tier 1) to complete it faster, will the change reflect immediately during the long running step 7 and the query will start receiving CPU & IO on high priority for that ongoing step as accordance to that newly assigned workload ? I do see change in Workload from Batch-Long to Batch Short in ViewPoint for that particular step but execution wise the step took same time. ( System was busy during that time but queires with low priority workloads were running on it)
Can you please explain on it ?
↧
Expediting Express Requests - comment by carrie
Steven,
SLES 10 and SLES 11 work in a similar way when it comes to the AWT work types different work messages will be assigned. In SLES 10, if a workload has a tactical enforcement priority it is considered an "expedited" workload. All of its requests are expedited. In SLES 11 any workload you place on the tactical tier is considered "expedited". All work messages that arrive on the AMPs from requests that come from an expedited workload will have a work type of either Work8, Work9, or Work10. It doesn't matter if there are reserve pools set up for those work types, tactical messages will still use work8/9/10.
Whether or not you choose to reserve AWTs for work8/9/10 is independent of the use of those work types by messages that come from expedited workloads. If you do set a reserve for tactical requests, that will hold back some number of AWTs to be used only for expedited requests. There is no extra overhead in using reserved AWTs that come from a reserve pool for work8 work type, for example. It is the same internal effort to grab an AWT from the work8 reserve pool as from the work00 reserve pool, or from the unassigned pool. In fact, the reserve pools are really logical constructs, in reality it's all one big pool of AWTs, with counters keeping track of reserves and who gets them and how many AWTs to leave in the pool to cover unused reserves.
If you do not have tactical work but you have defined reserves for work8/9/10, you will be holding back some number of AWTs that non-expedited work cannot use, so that could have a negative impact. That is the only downside I can think of for NOT reserving AWTs for expedited work if you find yourself running out of AWTs or ever in flow control. And even if you don't have tactical requests, as this posting above describes, if you have defined at least 2 reserves, parsing express requests can run there as well, so that could help you out.
I would suggest you talk to CS or the GSC before adding AWTs, just to make sure someone looks at your hardware and other details and make sure you won't run out of memory. It's a conservative step to take, and the right thing to do.
Thanks, -Carrie
↧
↧
How Resources are Shared in the SLES 11 Priority Scheduler - comment by carrie
Sachin,
If you move a query to a different workload in the middle of a step, the new priority will take effect immediately. If you don’t think giving a higher priority to the query is as effective as you would like, there are a couple of things you can check:
1. Check the global weight of both the Timeshare workload the query came from and the SLG Tier workload it was moved to. I have seen several cases where the SLG Tier workload was not given a high enough allocation percent, and actually ended up with a lower run-time priority than a workload in Timeshare. Although that is unlikely for an SLG Tier workload to have a lower global weight than a Timeshare Low workload, if Batch Long was the only workload active in Timeshare, it would get all of the resources that flow into Timeshare from the Remaining above, and so Batch Long could be running at a higher priority than the SLG Tier workload. The orange book explains how to calculate global weight, and there is another posting on my blog explaining global weight.
2. Check concurrency within the SLG Tier workload. SLG Tier workload allocatons are shared among all active requests within the workload. So expect a query to run much faster if it is the only request active there, and slower if there are 10 or 20 other active queries in the workload.
If this is a chronic problem, you could increase the allocation percent of the SLG Tier workload so it has a higher priority. Or reduce the concurrency of work running there, by adding a workload throttle.
Thanks, -Carrie
↧
Statistics Collection Recommendations – Teradata 14.10, 15.0 and 15.10 - comment by sri krishna C
Hi Carrie,
Need a clarification on how skipping stats works.
1. We had a traditional automated stats collection process where 10% was crietieria which we replaced with new feature of skipping process
question: Once we enable this skip process does 10% of change in data holds any value?
(reason our weekend stats collection is still based on 10 %)
2. After enabling this feature do we still need sample stats ??
can you direct me to an orange book or an article how the extrapolation works once this feature is enaabled
..
Happy Holidays
↧
Statistics Collection Recommendations – Teradata 14.10, 15.0 and 15.10 - comment by sri krishna C
Also one more question how is this stats skip related to stats_mngr process ?
↧
Christer,
There is no problem collecting sampled statstics on some columns and full statistics on others on the same table. Full stats are often a better choice when there is noteable skew in the column, as your example below points out. If you are sampling at a low percent, or even a moderate percent, it may be difficult to properly capture the degree of skew or the percent of null values.
Whatever statistics strategy is resulting in the best plans is usually the right action to take.
Thanks, -Carrie