programmer group

Python 3 Network Crawler Actual Warfare-19. Basic Principles of Agent

Last article: Python 3 Web Crawler Actual Warfare – 18, Session and CookiesNext article: Python 3 Web Crawler Actual Warfare – 20, Using Urllib: Sending Requests We often encounter this situation in the process of crawling. At first, the crawler works normally and grabs data normally. Everything looks so good. However, a cup of tea may make mistakes, such as 403 Forbidden. When we open the web page, we may see that “your IP access frequency is too high.” “Gao” is a hint. The reason for this phenomenon...

programmer group

Flask Notes: Flask. ext. Sqlalchemy or flask_sqlalchemy

Recently, the flask framework of Python has been introduced. The Sqlalchemy library is officially recommended as a tool to interact with the database. At that time, I found a problem while reading some articles and materials on the network. In addition, Sqlalchemy and flask_sqlalchemy are two different things. The latter is the encapsulation of Sqlalchemy in the framework of flask. There are many professional introductory documents on the internet. But here, as a technician who has been developing the back end for many years, I recommend...

programmer group

Use of Python async with and async for

There are few Chinese materials about async with and for online. I will translate the official statement of PEP 492. Async with Async Context Manager Asynchronous context manager refers to a context manager that can pause execution at the enter and exit methods. In order to achieve this function, two new methods need to be added: _aenter_ and _aexit_. Both methods return a value of awaitable type. One way to use an asynchronous context manager is to: class AsyncContextManager: async def __aenter__(self): await log('entering...

programmer group

Notes on Data Analysis Using Python--Basic Knowledge of Python

Read SeanCheney The blogger uploaded the article with a feeling, hereby write a note, such as invading must be deleted. Chapter 2 The Foundation of Python Grammar, IPython and Jupyter Notebooks 1. A cyclic initialization method for dictionaries data = {i : np.random.randn() for i in range(7)} Out[7]: {0: -0.20470765948471295, 1: 0.47894333805754824, 2: -0.5194387150567381, 3: -0.55573030434749, 4: 1.9657805725027142, 5: 1.3934058329729904, 6: 0.09290787674371767} 2. Jupyter Notebooks Tab can be used to complement, such as...

programmer group

six usage of python

Links to the original text: https://www.jianshu.com/p/62a6e3f2d1ca six usage of python 22017.07.05 15:05:24 words 1365 reading 12462 Preface Recently, in writing a Django project, I saw the use of six in the underlying functions of form, so I'll learn about it. Also hope that these things can better help you learn some advanced usage of Django and understand some source code of django. By writing this article, I hope to help myself understand some...

programmer group

Data structure comparison in Python

List: list = [val1, val2] 1. Each element in the list is variable, orderly, and indexed.Variability means that you can add, delete, and modify each element. The elements in the list can be any object in Python. (1) The method of creating lists: One is to assign a set of values separated by commas directly by square brackets []. newlist=[1,2,3] print(newlist) #[1, 2, 3] Print (type (newlist)#View type: <class'list'> 2. Call list function to define. tup=('q','w','e') ...

programmer group

Flask Notes: Flask Interaction with Databases

Flask-sqlalchemy is officially recommended, but I personally don’t recommend it here. Because background programmers prefer to use native SQL language to deal with database problems. One of the more important reasons is that when dealing with some operation and maintenance problems after the project goes online, if you can look at SQL statements, you can find problems faster. Second, flask-sqlalchemy encapsulates a layer on the basis of sqlalchemy, which is certainly less efficient from the compilation point of view (the impact of...

programmer group

Using Celery to Complete Asynchronous and Timing Tasks

0917 self-summary Use of Celery I. Official Documents Celery official website: http://www.celeryproject.org/ Celery official document in English: http://docs.celeryproject.org/en/latest/index.html Chinese version of Celery official document: http://docs.jinkan.org/docs/celery/ II. Celery Architecture Celery's architecture consists of three parts: message broker, worker and task result store. Message Middleware Celery itself does not provide message services, but it can easily integrate with message middleware provided by third parties....

programmer group

Millet Merchandise and Tencent Recruitment Multithread Crawling

The blog is reproduced from: https://www.cnblogs.com/LXP-Never/p/11378709.html.   Catalog _Millet App Store Grab (Multithread)_Tencent Recruitment Data Grabbing (Ajax)   Application scenario 1. Multiprocess: CPU-intensive programs2. Multithreading: Crawler (Network I/O), Local Disk I/O   Review of Knowledge Points queue # Import module from queue import Queue # Use q = Queue() q.put(url) q.get() # Blocking when queues are empty q.empty() # Determine whether the queue is empty. True/False   Thread module #...