一个Redis Cache实现(V2)

更新说明

 

  • V2:缓存锁改用redis本身的setnx,原来用python的set不够安全。逻辑上也有少许修改——在第一次请求时如果没有缓存则等待,原来是不等待直接返回None,后续请求则与原来相同,直接返回旧值不等待。redis_cached装饰器的db参数变成函数,以便于动态创建的redis对象使用。

需求

 

应用中需要通过HTTP调用远程的数据,但是这个获取过程需要执行较长时间,而且这个数据本身的变化也不频繁,这种情况最适合用一个cache来优化。

前两年在做短链接实现的时候,曾经用最好的语言PHP做过一个Redis cache实现《一个简单的Redis应用(修订版)》,但那个毕竟是一个特定的实现,而且我现在需要的是python版。

这次的目标是需要实现一个比较通用的cache,支持各种数据类型,有超时更新机制,超时更新需要有锁(防止前文那个例子里发生过的问题)。

代码(py3)

 

#!/usr/bin/env python
# -*- coding: utf-8 -*-
# python 3 required
from datetime import datetime
from functools import wraps
from traceback import format_exc
import hashlib
import pickle
import logging

from redis import StrictRedis as Redis

__author__ = 'raptor'

logger = logging.getLogger(__name__)

class RedisCache(object): MAX_EXPIRES = 86400 LOCK_EXPIRES = 60 SERIALIZER = pickle

<span class="hljs-function" style="box-sizing: border-box;"><span class="hljs-keyword" style="box-sizing: border-box;">def</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(153, 0, 0);">__init__</span><span class="hljs-params" style="box-sizing: border-box;">(<span class="hljs-keyword" style="box-sizing: border-box;">self</span>, name, host=<span class="hljs-string" style="box-sizing: border-box; color: rgb(221, 17, 68);">&#39;localhost&#39;</span>, port=<span class="hljs-number" style="box-sizing: border-box; color: rgb(0, 128, 128);">6379</span>, db=<span class="hljs-number" style="box-sizing: border-box; color: rgb(0, 128, 128);">0</span>, max_expires=MAX_EXPIRES)</span></span>:
    <span class="hljs-keyword" style="box-sizing: border-box;">self</span>.db = Redis(host=host, port=port, db=db)
    <span class="hljs-keyword" style="box-sizing: border-box;">self</span>.name = name
    <span class="hljs-keyword" style="box-sizing: border-box;">self</span>.max_expires = max_expires

<span class="hljs-function" style="box-sizing: border-box;"><span class="hljs-keyword" style="box-sizing: border-box;">def</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(153, 0, 0);">_getkey</span><span class="hljs-params" style="box-sizing: border-box;">(<span class="hljs-keyword" style="box-sizing: border-box;">self</span>, *keys)</span></span>:
    <span class="hljs-keyword" style="box-sizing: border-box;">return</span> <span class="hljs-string" style="box-sizing: border-box; color: rgb(221, 17, 68);">&quot;%s:%s&quot;</span> % (<span class="hljs-keyword" style="box-sizing: border-box;">self</span>.name, <span class="hljs-string" style="box-sizing: border-box; color: rgb(221, 17, 68);">&quot;:&quot;</span>.join(keys))

<span class="hljs-function" style="box-sizing: border-box;"><span class="hljs-keyword" style="box-sizing: border-box;">def</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(153, 0, 0);">_get_data</span><span class="hljs-params" style="box-sizing: border-box;">(<span class="hljs-keyword" style="box-sizing: border-box;">self</span>, key)</span></span>:
    result = <span class="hljs-keyword" style="box-sizing: border-box;">self</span>.db.get(key)
    <span class="hljs-keyword" style="box-sizing: border-box;">return</span> None <span class="hljs-keyword" style="box-sizing: border-box;">if</span> result == b<span class="hljs-string" style="box-sizing: border-box; color: rgb(221, 17, 68);">&#39;None&#39;</span> <span class="hljs-keyword" style="box-sizing: border-box;">else</span> result

<span class="hljs-function" style="box-sizing: border-box;"><span class="hljs-keyword" style="box-sizing: border-box;">def</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(153, 0, 0);">get</span><span class="hljs-params" style="box-sizing: border-box;">(<span class="hljs-keyword" style="box-sizing: border-box;">self</span>, *keys)</span></span>:
    result = <span class="hljs-keyword" style="box-sizing: border-box;">self</span>._get_data(<span class="hljs-keyword" style="box-sizing: border-box;">self</span>._getkey(*keys))
    <span class="hljs-keyword" style="box-sizing: border-box;">return</span> <span class="hljs-keyword" style="box-sizing: border-box;">self</span>.SERIALIZER.loads(result) <span class="hljs-keyword" style="box-sizing: border-box;">if</span> result is <span class="hljs-keyword" style="box-sizing: border-box;">not</span> None <span class="hljs-keyword" style="box-sizing: border-box;">else</span> result

<span class="hljs-function" style="box-sizing: border-box;"><span class="hljs-keyword" style="box-sizing: border-box;">def</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(153, 0, 0);">set</span><span class="hljs-params" style="box-sizing: border-box;">(<span class="hljs-keyword" style="box-sizing: border-box;">self</span>, keys, value, ex=None)</span></span>:
    k = <span class="hljs-keyword" style="box-sizing: border-box;">self</span>._getkey(*keys)
    v = <span class="hljs-keyword" style="box-sizing: border-box;">self</span>.SERIALIZER.dumps(value)
    <span class="hljs-keyword" style="box-sizing: border-box;">self</span>.db.set(k, v, ex=ex)

<span class="hljs-function" style="box-sizing: border-box;"><span class="hljs-keyword" style="box-sizing: border-box;">def</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(153, 0, 0);">delete</span><span class="hljs-params" style="box-sizing: border-box;">(<span class="hljs-keyword" style="box-sizing: border-box;">self</span>, *keys)</span></span>:
    <span class="hljs-keyword" style="box-sizing: border-box;">self</span>.db.delete(<span class="hljs-keyword" style="box-sizing: border-box;">self</span>._getkey(*keys))

@staticmethod
<span class="hljs-function" style="box-sizing: border-box;"><span class="hljs-keyword" style="box-sizing: border-box;">def</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(153, 0, 0);">build_key</span><span class="hljs-params" style="box-sizing: border-box;">(name, *args, **kwargs)</span></span>:
    m = hashlib.md5()
    m.update(name.encode(<span class="hljs-string" style="box-sizing: border-box; color: rgb(221, 17, 68);">&#39;utf-8&#39;</span>))
    m.update(pickle.dumps(args))
    m.update(pickle.dumps(kwargs))
    <span class="hljs-keyword" style="box-sizing: border-box;">return</span> m.hexdigest()

<span class="hljs-function" style="box-sizing: border-box;"><span class="hljs-keyword" style="box-sizing: border-box;">def</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(153, 0, 0);">cached</span><span class="hljs-params" style="box-sizing: border-box;">(<span class="hljs-keyword" style="box-sizing: border-box;">self</span>, key, func, ex=None)</span></span>:
    <span class="hljs-keyword" style="box-sizing: border-box;">if</span> ex is <span class="hljs-symbol" style="box-sizing: border-box; color: rgb(153, 0, 115);">None:</span>
        ex = <span class="hljs-keyword" style="box-sizing: border-box;">self</span>.max_expires
    min_ttl = <span class="hljs-keyword" style="box-sizing: border-box;">self</span>.max_expires - ex  <span class="hljs-comment" style="box-sizing: border-box; color: rgb(153, 153, 136); font-style: italic;"># ex &lt;= 0 : force refresh data</span>
    key = <span class="hljs-string" style="box-sizing: border-box; color: rgb(221, 17, 68);">&quot;:&quot;</span>.join([<span class="hljs-keyword" style="box-sizing: border-box;">self</span>.name, key])
    result = <span class="hljs-keyword" style="box-sizing: border-box;">self</span>._get_data(key)
    lock_key = <span class="hljs-string" style="box-sizing: border-box; color: rgb(221, 17, 68);">&quot;:&quot;</span>.join([<span class="hljs-string" style="box-sizing: border-box; color: rgb(221, 17, 68);">&quot;__lock__&quot;</span>, key])
    <span class="hljs-keyword" style="box-sizing: border-box;">if</span> <span class="hljs-keyword" style="box-sizing: border-box;">self</span>.db.set(lock_key, <span class="hljs-number" style="box-sizing: border-box; color: rgb(0, 128, 128);">1</span>, ex=<span class="hljs-keyword" style="box-sizing: border-box;">self</span>.LOCK_EXPIRES, nx=True):
        <span class="hljs-symbol" style="box-sizing: border-box; color: rgb(153, 0, 115);">try:</span>
            ttl = <span class="hljs-keyword" style="box-sizing: border-box;">self</span>.db.ttl(key)
            <span class="hljs-keyword" style="box-sizing: border-box;">if</span> ttl is None <span class="hljs-keyword" style="box-sizing: border-box;">or</span> ttl &lt; <span class="hljs-symbol" style="box-sizing: border-box; color: rgb(153, 0, 115);">min_ttl:</span>
                result = func()
                <span class="hljs-keyword" style="box-sizing: border-box;">if</span> result is <span class="hljs-keyword" style="box-sizing: border-box;">not</span> <span class="hljs-symbol" style="box-sizing: border-box; color: rgb(153, 0, 115);">None:</span>
                    result = <span class="hljs-keyword" style="box-sizing: border-box;">self</span>.SERIALIZER.dumps(result)
                <span class="hljs-keyword" style="box-sizing: border-box;">self</span>.db.set(key, result, ex=<span class="hljs-keyword" style="box-sizing: border-box;">self</span>.max_expires)
        <span class="hljs-symbol" style="box-sizing: border-box; color: rgb(153, 0, 115);">finally:</span>
            <span class="hljs-keyword" style="box-sizing: border-box;">self</span>.db.delete(lock_key)
    elif result is <span class="hljs-symbol" style="box-sizing: border-box; color: rgb(153, 0, 115);">None:</span>
        <span class="hljs-keyword" style="box-sizing: border-box;">for</span> i <span class="hljs-keyword" style="box-sizing: border-box;">in</span> range(<span class="hljs-keyword" style="box-sizing: border-box;">self</span>.LOCK_EXPIRES * <span class="hljs-number" style="box-sizing: border-box; color: rgb(0, 128, 128);">10</span>):
            time.sleep(<span class="hljs-number" style="box-sizing: border-box; color: rgb(0, 128, 128);">0</span>.<span class="hljs-number" style="box-sizing: border-box; color: rgb(0, 128, 128);">1</span>)
            <span class="hljs-keyword" style="box-sizing: border-box;">if</span> <span class="hljs-keyword" style="box-sizing: border-box;">not</span> <span class="hljs-keyword" style="box-sizing: border-box;">self</span>.db.exists(lock_key):
                <span class="hljs-keyword" style="box-sizing: border-box;">break</span>
        result = <span class="hljs-keyword" style="box-sizing: border-box;">self</span>._get_data(key)
    result = <span class="hljs-keyword" style="box-sizing: border-box;">self</span>.SERIALIZER.loads(result) <span class="hljs-keyword" style="box-sizing: border-box;">if</span> result is <span class="hljs-keyword" style="box-sizing: border-box;">not</span> None <span class="hljs-keyword" style="box-sizing: border-box;">else</span> None
    <span class="hljs-keyword" style="box-sizing: border-box;">return</span> result

def redis_cached(get_db, ex=None): def decorator(fn): @wraps(fn) def wrapper(*args, **kwargs): try: key = RedisCache.build_key(fn.__name__, *args, **kwargs) return get_db().cached(key, lambda: fn(*args, **kwargs), ex) except: logger.error(format_exc()) return None return wrapper return decorator

用法

 

RedisCache本身也可以当一个Redis数据库对象使用,比如:

db = RedisCache('tablename', max_expires=3600)  #  tablename是一个是自定义的key前缀,可以用于当作表名使用。
# 最大超时时间(max_expires)仅供cached使用,使用set时,如果不指定超时时间则永不超时
db.set('aaa', {'key': 1234}, 7200)  # value可以是任何可序列化数据类型,比如字典,不指定超时则永不超时
db.get('aaa')['key']  # 结果为1234
db.delete('aaa')

但这个不是重点,重点是cached功能。对于慢速函数,加上db.cached以后,可以对函数调用的结果进行cache,在cache有效的情况下,大幅提高函数在反复调用时的性能。

下面是一个例子,具体见代码中的注释:

db = RedisCache('tablename')

def func(url, **kwargs): result = requests.get("?".join([url, urlencode(kwargs)])) return result

url = 'https://www.baidu.com/s' t = time() func(url, wd="测试") print(time()-t) # 较慢 t = time() db.cached('test_cache', lambda: func(url, wd="测试"), 10) print(time()-t) # 第一次运行仍然较慢 t = time() db.cached('test_cache', lambda: func(url, wd="测试"), 10) print(time()-t) # redis里读取cache很快 sleep(11) # 等待到超时 t = time() db.cached('test_cache', lambda: func(url, wd="测试"), 10) print(time()-t) # 超时后会再次执行func更新cache t = time() db.cached('test_cache_new', lambda: func(url, wd="新的测试")) print(time()-t) # 不同的调用参数用不同的keycache t = time()

因为对于不同的函数调用参数,函数可能有不同的返回结果,所以应该用不同的key进行cache。为简单起见,可以把函数签名做一个HASH,然后以此为KEY进行cache。最后把这个操作做成一个decorator,这样,只需要给函数加上这个decorator即可自动提供所需要的cache支持。

最终的简单用法如下:

db = RedisCache('tablename')

@redis_cached(lambda: db, 10) def func(url, **kwargs): result = requests.get("?".join([url, urlencode(kwargs)])) return result

t = time() func(url, wd="测试") print(time()-t) t = time() func(url, wd="测试") print(time()-t) sleep(11) t = time() func(url, wd="测试") print(time()-t) t = time() func(url, wd="新的测试") print(time()-t)

是不是简单得多了。

推送到[go4pro.org]