Eurusan

Methodology

• event-based profiling
  • language
    • python
      • sys.setprofile python_[call|return|exception] c_[call|return|exception]
  • event-based profiler | tracing profiler
• statistical profiling
  • less data
  • less effect performance (profiler will have a negative effect)
  • Oprofile
  • statprof
• vector
  • operation time
• bottleneck
  • I/O
    • disk
    • network
    • sql
  • memory
    • no enough memory
    • loading data, but not release
    • avg | max |min
  • slow*times
  • noCache*times
• Running Time Complexity, RTC, big O notation
• practice
  • Regression Testing ()
• profiling before optimization
• coverage
• unitest
  • Don’t let the modification destroy the normal operation of the program.
• framework
  • repeat manual work.
  • multi Progress.
• be patient.
• collect everthing
  • system log
  • user defined log
  • system Snapshot
• Pretreatment
  • ETL
• profier
  • cProfile
    • info
      • ncalls
      • tottime: find bad loop
      • percall
        • n: call times
        • n/m: total-times/recursion-times
      • cumtime: function+=sub-function-call…
      • percall: tottime/ncalls
      • filename:lineno(function): location
    • api
      • run:
      • runctx: save result into file
      • Profile
  • pstats
    • prof:=cProfile
    • pstats(prof)
    • profile.Stats+cProfile.Profile
  • line_profiler
    • kernprof -l file.py -> file.py.lprof
    • Line
    • Hints
    • Time
    • Per hit
    • % time
• Visualization
  • KCacheGrind/pyprof2calltree: cProfile->KCacheGrind
  • RunSnakeRUn: cProfile onlypyhon2
• method
  • memoization
    • default agrument + static argument + cache: fastest
  • lookup table
  • dis + complie
    • dis.disassemble(compile(text,'<string>','exec'))
    • compare Instruction Set
  • generator expression
  • ctypes
    • cdll
  • stirng
    • join && +=
  • collection.deque
  • while 1 better than while True
  • namedtuple better than class instance
  • nultiple assignments is slow.
• multithreading vs. multiprocessing
  • concurrency
    • time slicing
  • parallelism
    • CUDA
    • OpenCL
  • multithreading
    • package
      • thread
        • allocate_lock
          • acquire
          • release
          • locked
      • threading
        • join: pending on thrading task
        • conmmunation
          • event
            • is_set
            • set
            • clear
            • wait
    • pros
      • continued response
      • faster (use multi core)
      • less memory
      • simple state share
      • simple thread communication
    • cons
      • race condition
      • A crash in a thread takes down the whole process.
      • deadlock
  • multiprocessing
    • package
      • multipricessing
        • multiprocessing.Process(target=foo,args("bar"))
        • p.terminate()
        • p.start()
        • p.join()
        • p.exitcode
        • Pool
          • apply
          • apply_async
            • ApplyResult
              • get()
        • communation
          • Queue
          • Pipe
          • Event
    • prod
      • multi core
      • independent memory space
      • killable
      • GIL
    • cons
      • more memory
      • IPC complex than multithreading
      • share data complex
• common
  • PyPy
    • speed JIT
    • memory less
    • sandboxing
    • stackless
  • cStringIO
  • Cython
• number crunching
  • Numba: numba jit, numpy
  • Parakeet
  • pandas