샐 러 리 노트 [샐 러 리 에 대해 이야기 할 때 알려 져 야 할 것들]

제 블 로그 에서 돌 아 왔 습 니 다. 소통 많이 해 주세요.
http://litaotao.github.io/hello-celery/
1. hello, celery

three questions:
- what is celery?
- why do we need celery?
- what features does celery have?

1.1 what is celery?

First , celery is a distributed task queue [ a distributed system to process messages].

Second , it provides operations with the tools required to maintain such a system.

flower

jobtastic

Third , it was built for Dijango Apps in the early, but scaled since version 0.8.

Fourth , it focus on real-time processing, while also supporting task scheduling.

1.2 why do we need celery?

run something in the background.

run something after the web request is done.

making something is done by executing it asynchronously and using retries.

schedule periodic task.

distributed computing.

parallel executing.

1.3 what features does celery have?

simple : write in python, easily configure.

high available : retry mechanism, independent broker HA.

fast : process millions of tasks one minute.

flexible : Almost every part of Celery can be extended or used on its own, Custom pool implementations, serializers, compression schemes, logging, schedulers, consumers, producers, autoscalers, broker transports and much more.

monitoring : A stream of monitoring events is emitted by workers and is used by built-in and external tools to tell you what your cluster is doing – in real-time.

auto reloading : In development workers can be configured to automatically reload source code as it changes.

workflows : Simple and complex workflows can be composed using a set of powerful primitives we call the “canvas”, including grouping, chaining, chunking and more.

auto scaling : Dynamically resizing the worker pool depending on load, or custom metrics specified by the user, used to limit memory usage in shared hosting/cloud environments or to enforce a given quality of service.

Time & Rate Limits : You can control how many tasks can be executed per second/minute/hour, or how long a task can be allowed to run, and this can be set as a default, for a specific worker or individually for each task type.

Scheduling : You can specify the time to run a task in seconds or a datetime, or or you can use periodic tasks for recurring events based on a simple interval, or crontab expressions supporting minute, hour, day of week, day of month, and month of year.

Resource leak protection :

User components :

2 how does celery work?
2.1 some terms in message queue and celery

celery task : a serialized task info which will be scheduled to done by celery,

it contains some meta data, like bellow:

{"id": "4cc7438e-afd4-4f8f-a2f3-f46567e7ca77",
 "task": "celery.task.PingTask",
 "args": [],
 "kwargs": {},
 "retries": 0,
 "eta": "2009-11-17T12:30:56.527191"
}

producer : the one who send task messages to the broker,

it is a celery client.

consumer : fetch task messages from queues and process them.

broker : the one who receives task messages from producer and route different

to different queues.

exchange : exchange, or a router, the one who really do message receiving from producer

dispatching message to queues. two things should be taken in mind about exchange:

one message from producer can be routed to zero or more queues;

you can set route rules for the exchange, called bindings;

queue : buffer for store task messages.

2.2 the celery workflow
2.3 the workflow deployment
3. install celery
two possible ways to install celery.

install the latest stable celery and its dependences

chenshan@mac007:~/Desktop$sudo pip install -U celery
...
...
...
Installing collected packages: celery, kombu, billiard, amqp

the celery dependences info is listed here: Does Celery have many dependencies

install celery with bundles

Celery also defines a group of bundles that can be used to install Celery and the dependencies for a given feature.
You can specify these in your requirements or on the pip comand-line by using brackets. Multiple bundles can be specified by separating them by commas. for example, using the command pip install "celery[librabbitmq,redis,auth,msgpack]" to install celery with librabbitmq, redis, auth, msgpack libriaries.
all bundles command can be find here: bundles
참고 문서

celery official site

http://abhishek-tiwari.com/post/amqp-rabbitmq-and-celery-a-visual-guide-for-dummies

Async Celery by Example: Why and How

How Instagram Feeds Work: Celery and RabbitMQ

stackoverflow: Celery parallel distributed task with multiprocessing

Scaling Celery Sending Tasks To Remote Machines!

Celery Messaging at Scale at Instagram

CELERY - BEST PRACTICES

CELERY - BEST PRACTICES slide

CELERY - 최고의 가격 중국어

celery 가 구 덩이 를 밟 는 과정 에서 Query set 의 게 으 름 로드

에 대해 이야기 합 니 다.

stackoverflow: Send log messages from all celery tasks to a single file

이 내용에 흥미가 있습니까?

현재 기사가 여러분의 문제를 해결하지 못하는 경우 AI 엔진은 머신러닝 분석(스마트 모델이 방금 만들어져 부정확한 경우가 있을 수 있음)을 통해 가장 유사한 기사를 추천합니다:

셀러리가 작업을 마칠 때까지 기다립니다.

Python 애플리케이션을 배포할 때 셀러리를 다시 시작해야 하는 경우가 있습니다. 오래 실행되는 작업이 있는 경우 해당 작업을 종료하지 않고 작업이 완료될 때까지 기다리십시오. 이 기능은 새 작업 수락을 일시 중지...

텍스트를 자유롭게 공유하거나 복사할 수 있습니다.하지만 이 문서의 URL은 참조 URL로 남겨 두십시오.

CC BY-SA 2.5, CC BY-SA 3.0 및 CC BY-SA 4.0에 따라 라이센스가 부여됩니다.

샐 러 리 노트 [샐 러 리 에 대해 이야기 할 때 알려 져 야 할 것들]

좋은 웹페이지 즐겨찾기