How to optimize and structure the construction of medium and large websites
Source: Shangpin China |
Type: website encyclopedia |
Time: November 25, 2015
The above website architecture is widely used in medium and large scale Beijing website construction This article analyzes the mainstream technologies and solutions used from each layer of the architecture, which is helpful for new website operation and maintenance friends to further understand the website architecture and form a set of architecture concepts.
First layer: CDN
The domestic network is mainly distributed in South China Telecom and North China Unicom, which causes a large problem of cross regional access delay. For websites with a certain number of visits, adding the CDN (content distribution network) layer can effectively improve this phenomenon, and it is also the best choice for website acceleration. CDN caches website pages to nodes distributed nationwide, and users can obtain data from the nearest computer room when visiting, which greatly reduces the path of network access. If you want to build your own CDN, you are not recommended to do so. Why? In fact, to put it bluntly, don't stop the operation and maintenance. The deployment of CDN architecture is not complicated, but there are many factors that affect the effect. Later management and maintenance are also complex. It is not easy to achieve the expected effect. It is a thankless job. Finally, the boss still feels that your ability is insufficient. It is recommended to find a company that specializes in CDN. The cost is not expensive, it has the ability to resist traffic attacks, and the effect is also very good. There is also much less to do in O&M, so why not do it!
Layer 2: Reverse proxy (web cache)
If the CDN does not cache the data to be requested, send a request to this layer. Configure the caching function (local) in the proxy server, and the proxy server will find out whether there is data requested by the CDN in the local cache. If there is data, it will directly return it to the CDN. If there is no data, it will request the back-end load balancer and then forward it to the WEB server to return the data to the proxy server, The proxy server sends the result to the CDN. The proxy server generally caches static pages that do not change frequently, such as image, js, css, html, etc. The mainstream cache software includes Squid, Varnish, and Nginx.
Layer 3: load balancing
Load balancing is used for websites with large traffic, because it is the best way to solve the performance bottleneck of a single server. The reverse proxy forwards the request to the load balancer. The load balancer selects the backend according to the algorithm (rotation training, load conditions, etc.) and hands it to the backend WEB service for processing. After the WEB service processing is completed, the data is directly returned to the reverse proxy server. Load balancing reasonably distributes requests to multiple back-end WEB servers to reduce the concurrent load of a single server and ensure service availability. The mainstream load balancing software includes LVS, HAProxy, and Nginx.
Layer 4: Web services
Web services process user requests. The efficiency of Web service processing directly affects the access speed. To avoid slow access caused by this factor, it should be optimized to make Web services play their best role. Common web services include Apache and Nginx.
Apache optimization:
1) . mod_deflate compression module
Check whether to load:
# apachectl M |grep deflate
If it is not installed, use apxs to compile it:
#/usr/local/apache/bin/apxs c I A apache source directory/modules/mod_detemplate c
Deflate configuration parameters:
DeflateCompressionLevel6 # Compression level (1-9). The higher the value, the higher the efficiency, and the higher the CPU consumption SetOutputFilterDEFLATE # Enable compression AddOutputFilterByTypeDEFLATE text/html text/plain text/xml # Compression type AddOutputFilterByTypeDEFLATE css js html htm xml php 2) . mod_expires cache module
Check whether to load:
# apachectl M |grep expires
If it is not installed, use apxs to compile it:
#/usr/local/apache/bin/apxs c I A apache source directory/modules/mod_expires c
Then enable the module in httpd.conf: LoadModule expires_module modules/mod_expires.so
The cache mechanism has three usages: global, directory, and virtual host
Global configuration, add at the end of the configuration file:
ExpiresActiveon # Enable validity control, which will automatically clear the expired cache, and then obtain new ExpiresDefault "accessplus 1 days" # By default, documents in any format expire in 1 day ExpiresByTypetext/html "access plus 12 months" ExpiresByTypeimage/jpg "access plus 12 months" # jpg Format Picture Cache December
3) . Selection and optimization of working mode
Apache has two common working modes, worker and prefork. The default is worker, which is a hybrid MPM (multiprocessing module). It supports multiple processes and threads. Threads process requests, so it can process more requests and improve concurrency. The system resource overhead is also smaller than the process based MPM. Because threads use process memory space, A process crash will cause its underlying thread to crash. Prefork is a non threaded MPM, and processes occupy more system resources than workers. Because processes process connections, they are also more stable in work efficiency than workers. You can view the current working mode through apache2 l, and use the - with mpm parameter to specify the working mode when compiling. The processing capacity can be improved by selecting different working modes according to your own business needs and adding appropriate parameters related to the working mode.
Nginx optimization:
1) . gzip compression module
http { …… gzip on; gzip_min_length 1k; # The minimum number of bytes allowed to be compressed on a page is 0 by default. Most pages are compressed, and those smaller than 1k may be counterproductive gzip_buffers 4 16k; # The size of the memory requested by gzip is 4 times the size of the data gzip_http_version 1.0; # Identify http protocol version gzip_comp_level 2; # Compression level: 1 has the smallest compression ratio, the fastest processing speed, 9 has the largest compression ratio, and the slowest processing speed gzip_types text/plainapplication/x-javascripttext/css application/xml image/jpg; # Compressed data type gzip_vary on; # Judge whether compression is required according to the http header of the client }
2) . expires cache module
server { location ~ .*. (gif | jpg | png | bmp | swf) $# Cache data suffix type { expires 30d; # Use the expires cache module to cache to the client for 30 days } location ~ .*. ( jsp|js|css)?$ { expires 1d; } }
3) . fastcgi optimization
Nginx does not support calling or parsing dynamic programs (php) directly. You must start the php fpm process through fastcgi (Universal Gateway Interface) to parse php scripts. That is to say, the user requests to go to nginx first, and nginx then sends the dynamic parsing to fastcgi, and fastcgi starts php fpm to parse the php script. Therefore, it is necessary to optimize the parameters of fastcgi and php fpm appropriately.
http { …… fastcgi_cache_path/usr/local/nginx/fastcgi_cache levels=1:2 keys_zone=TEST:10m inactive=5m; #FastCGI cache specifies a file path, directory structure level, key area storage time and inactive deletion time fastcgi_connect_timeout 300; # Specify the timeout for connecting to the backend FastCGI fastcgi_send_timeout 300; # Specify the timeout for sending requests to FastCGI fastcgi_read_timeout 300; # Specify the timeout for receiving FastCGI replies fastcgi_buffer_size 64k; # Specifies the size of the buffer needed to read the first part of the FastCGI response fastcgi_buffers 4 64k; # Specify how many boxes and buffers are needed locally to buffer FastCGI response requests fastcgi_busy_buffers_size 128k; fastcgi_temp_file_write_size 128k; # Indicates the size of the data block used when writing cache files. The default value is twice that of fastcgi_buffers fastcgi_cache TEST; # Enable the fastcgi_cache cache and specify a TEST name fastcgi_cache_valid 200 302 1h; # Specify the cache of 200 and 302 response codes for 1 hour fastcgi_cache_valid 301 1d; # Cache 301 response code for 1 day fastcgi_cache_valid any 1m; # Cache other responses for 1 minute { Php-fpm.conf configuration parameters:
Pm=dynamic # Two methods of controlling child processes (static and dynamic) Pm. max_children=5 # Maximum number of child processes surviving at the same time Pm. start_servers=2 # Number of processes created at startup Pm. min_spare_servers=1 # Minimum number of php fpm processes Pm. max_spare_servers=3 # Maximum number of php fpm processes
4) . proxy_cache local cache module
http { …… proxy_temp_path /usr/local/nginx/proxy_cache/temp; # Cache temporary directory proxy_cache_path /usr/local/nginx/proxy_cache/cache levels=1:2 keys_zone=one:10m inactive=1d max_size=1g; #The actual directory of the cache file. Levels define the hierarchical directory. 1:2 indicates that 1 is the primary directory and 2 is the secondary directory. Keys_zone stores metadata and allocates 10M memory space. Inactive means that the cache that has not been accessed in one day will be deleted. The default is 10 minutes. Max_size is the maximum allocated disk space server { listen 80; server_name 192.168.1.10; location / { proxy_cache one; # Call buffer #proxy_cache_valid 200 304 12h; # Different cache time can be set according to HTTP status code proxy_cache_valid any 10m; # Cache validity is 10 minutes } #To clear the URL cache, which network segment IP address is allowed to clear the cache (you need to install the third-party module "ngx_cache_purge"). To clear the URL cache, visit //192.168.1.10/purge/ file name location ~ /purge(/.*){ allow 127.0.0.1; allow 192.168.1.0/24; deny all; proxy_cache_purge cache_one$host$1$is_args$args; } }
Marketing website construction Summary of Shangpin China:
Enabling the compression module can save some bandwidth and increase the CPU processing on the WEB side. However, in the website architecture above, enabling the compression module on the WEB side does not work because the transmission to the upper layer is via the LAN. The user oriented architecture should be enabled. The WEB also does not need to enable the expires module. Because of the reverse proxy server and CDN, it cannot reach the user's browser and cannot be enabled.
If the reverse proxy uses nginx as the proxy, it can open the expires module to cache the static files to the user's browser. When the browser initiates a request, it first determines whether the local cache has the requested data, and then determines whether it has expired. If it does not, it can directly browse the cached data, even if the server resources have changed, Therefore, the expiration time should be set reasonably according to the business situation.
5. Use PHP cache to improve code execution efficiency
When a php program does not use a cache, every time it requests a php page, php will compile the code for the page, which means that repeated compilation will increase the server load. With the cache, the data after each compilation will be cached in the shared memory, and the compiled code in the buffer will be directly used for the next access, so as to avoid repeated compilation process and speed up its execution efficiency. Therefore, it is absolutely necessary for PHP websites to use cache! Mainstream PHP caches include eAccelerator and XCache
The fifth layer: static and dynamic separation
Dynamic static separation, as the name implies, is to separate dynamic pages and static pages to different servers for processing. For example, if the web is nginx, fastcgi can be deployed to a separate server to specifically parse php dynamic pages. Static pages are handled by nginx by default, and caching strategies can be implemented. Another example is a mall website, where there will be a large number of images. You can consider adding a file server group to send the requested images and uploaded images to the file server for processing. NFS is the mainstream file server, and there is a single point of failure. DRBD+HeartBeat+NFS can be deployed for high availability. If the pressure on a single server is too high, consider using distributed file systems, such as GlusterFS and MooseFS.
Layer 6: database cache
Using cache technology, cache the hot data into memory. If the requested data is in the cache, it will be returned directly. Otherwise, it will be fetched from the database and updated to the cache system to improve reading performance and reduce database pressure. Cache implementation includes local cache and distributed cache. Local cache is used to cache data in the local server memory or files. Distributed cache is to cache data in memory. It is distributed and can cache massive data with good scalability. Mainstream distributed cache systems include Memcached and Redis. Memcached has stable performance and fast speed. QPS can reach about 8w. If you want data persistence, choose Redis, whose performance is no lower than Memcached.
Layer 7: Database
This layer plays a leading role in the entire website architecture, directly determining the user experience, and relatively complex architecture optimization.
Core idea: reduce the request layer, let the front-end layer return the data requested by users as much as possible, and reduce the access frequency of the back-end server. The most important is the database layer.
Source Statement: This article is original or edited by Shangpin China's editors. If it needs to be reproduced, please indicate that it is from Shangpin China. The above contents (including pictures and words) are from the Internet. If there is any infringement, please contact us in time (010-60259772).
