First Throw A Famous Saying To The Town Building.
When we really need to start to really optimize our Python program, the first step we have to do is not blindly optimize, but to analyze our existing programs and discover the performance bottlenecks of the program. Sexual optimization. This will allow us to spend the most time and optimization to get the most out of the optimization.
This article focuses on the elegant use of Python’s built-in performance analyzer, and uses the author’s chemical kinetics program as an example for performance analysis practice, introduces the use of commonly used performance analysis visualization tools, and finally performs preliminary performance on Python programs. Optimization attempts.
About Performance Analysis
Performance analysis is to analyze the relationship between the code and the resources being used. It can help us analyze the running time to find the bottleneck of the running of the program, and also help us analyze the use of memory to prevent the occurrence of memory leaks.
The tool that helps us perform performance analysis is the performance analyzer, which is divided into two main categories: 1. event-based profiling 2. statistical profiling
Python Performance Analyzer
The most commonly used performance analysis tools in Python are a profiler, line_profiler, and memory_profiler. They help us analyze the performance of Python code in different ways. We focus here on Python’s built-in profiler and use it to help us analyze and optimize our programs.
From the analysis report results, we can get a lot of information: 1. A total of 197 function calls are monitored throughout the process, of which 192 are native calls (ie no recursive calls are involved) 2. The total execution time is 0.002 seconds 3. Results list The middle is sorted according to the standard name, that is, according to the way the string is printed (the number is also treated as a string) 4. In the list: – calls indicates the number of times the function is called (there are two values indicating that there are recursive calls, the total number of calls) / Native calls) – to time is the internal call time of the function (not including the time of other functions that it calls itself) – per-call is equal to to time / calls – cumtime cumulative call time, contrary to tottime, it contains the time of its own internal call function – Last column, file name, line number, function name
Python provides us with a number of interfaces that allow us to perform performance analysis flexibly, including two Profile classes for the cProfile module and the Stats class for the pstat module.
We can use these two classes to encapsulate the functionality of code analysis for flexible and reusable analysis elsewhere in the project.
Here is a brief summary of the common interfaces of Profile and Stats:
Enable(): Start collecting performance analysis data
Disable(): Stop collecting performance analysis data
Create_stats(): Stop collecting analytics data and create stats objects for collected data
Print_stats(): Create a stats object and print the analysis results
Dump_stats(filename): Write the result of the current performance analysis to a file (binary format)
Runcall(func, *args, **kwargs): collect performance analysis data of the called function func
The Stats class provided by the Stats pstats module helps us read and manipulate stats files (binary format) python import pstats p = pstats.Stats(‘stats.prof’) The Stats class can accept stats filenames or accept cProfile directly. The Profile object acts as a data source.
Strip_dirs(): Delete path information for all function file names in the report
Dump_stats(filename): Write the analysis data in stats to the file (the effect is the same as cProfile.Profile.dump_stats())
Sort_stats(*keys): Sorts the report list. The functions are sorted according to the parameters passed in. The keywords include calls, cumtime, etc. For specific parameters, see https://docs.python.org/2/library/profile.html #pstats.Stats.sort_stats
Reverse_order(): Reverse the current sort
Print_stats(*restrictions): Prints the information to standard output. *restrictions are used to control the form of the printed result. For example, (10, 1.0, “.*.py.*”) indicates the first 10 lines of the information for printing all py files.
With the above interface, we can use the parser to analyze our program more elegantly. For example, you can write a decorator with parameters, so that you can easily use any decorator to analyze any function in the project. Achieve their goals.
After the program finishes running, it will generate the analysis file of mkm_run.prof in the current path, and we can analyze this function through printing or visualization tools.
Performance analysis practice
Below I will practice by analyzing the methods in the MicroKineticModel class in my kinetics program, and use several commonly used performance analysis visualization tools to help analyze and perform preliminary optimization and efficiency comparison.
Note: The program of this test mainly includes numerical solution differential equation and Newton method to solve the solution of multivariate nonlinear equations. The formula derivation part of the program is completed by string operation.
Generate a performance analysis report
According to the above method, we modify the run method through the decorator to perform performance analysis, so that we can run the program as normal, but the difference is that the performance analysis report file will be generated in the current path.
After the seemingly normal run, we will generate an analysis report in the current path, mkm_run.prof, which is a binary file that we need to read using the interface of python’s pstats module.
We only sorted in descending order according to the accumulated time and output the first ten lines. The whole function only runs for 0.106 seconds. It can be seen that most of the time of the program is mainly in the process of solving the Newton method, and the process of obtaining the Jacobian Matrix is a major time-consuming part.
Although we can view the function call relationship through the command line, I don’t want to take the time to continue the analysis in the anti-human black and white box. Below I intend to use the visual visualization tool.