使用 alembic 管理数据库版本

This commit is contained in:
Chuangbo Li 2022-09-04 13:52:33 +08:00 committed by GitHub
parent a3828debd7
commit 9261232c56
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
13 changed files with 2409 additions and 19 deletions

View File

@ -30,10 +30,24 @@ pip install --upgrade poetry
### 2. 安装依赖
```bash
poetry update
poetry install
```
### 3. 运行
### 3. 修改配置
创建 `.env` 文件并填写数据库连接和 bot token 等参数。
```bash
cp .env.example .env
```
### 4. 初始化数据库
```bash
alembic upgrade head
```
### 5. 运行
```bash
python ./main.py
@ -50,4 +64,4 @@ python ./main.py
| [原神抽卡全机制总结](https://www.bilibili.com/read/cv10468091) | 本项目抽卡模拟器使用的逻辑 |
| [西风驿站 猫冬](https://bbs.mihoyo.com/ys/accountCenter/postList?id=74019947) | 本项目攻略图图源 |
| [Yunzai-Bot](https://github.com/Le-niao/Yunzai-Bot) | 本项使用的抽卡图片和前端资源来源 |
| [Crawler-ghhw](https://github.com/DGP-Studio/Crawler-ghhw) | 本项目参考的爬虫代码 |
| [Crawler-ghhw](https://github.com/DGP-Studio/Crawler-ghhw) | 本项目参考的爬虫代码 |

106
alembic.ini Normal file
View File

@ -0,0 +1,106 @@
# A generic, single database configuration.
[alembic]
# path to migration scripts
script_location = alembic
# template used to generate migration file names; The default value is %%(rev)s_%%(slug)s
# Uncomment the line below if you want the files to be prepended with date and time
# see https://alembic.sqlalchemy.org/en/latest/tutorial.html#editing-the-ini-file
# for all available tokens
# file_template = %%(year)d_%%(month).2d_%%(day).2d_%%(hour).2d%%(minute).2d-%%(rev)s_%%(slug)s
# sys.path path, will be prepended to sys.path if present.
# defaults to the current working directory.
prepend_sys_path = .
# timezone to use when rendering the date within the migration file
# as well as the filename.
# If specified, requires the python-dateutil library that can be
# installed by adding `alembic[tz]` to the pip requirements
# string value is passed to dateutil.tz.gettz()
# leave blank for localtime
# timezone =
# max length of characters to apply to the
# "slug" field
# truncate_slug_length = 40
# set to 'true' to run the environment during
# the 'revision' command, regardless of autogenerate
# revision_environment = false
# set to 'true' to allow .pyc and .pyo files without
# a source .py file to be detected as revisions in the
# versions/ directory
# sourceless = false
# version location specification; This defaults
# to alembic/versions. When using multiple version
# directories, initial revisions must be specified with --version-path.
# The path separator used here should be the separator specified by "version_path_separator" below.
# version_locations = %(here)s/bar:%(here)s/bat:alembic/versions
# version path separator; As mentioned above, this is the character used to split
# version_locations. The default within new alembic.ini files is "os", which uses os.pathsep.
# If this key is omitted entirely, it falls back to the legacy behavior of splitting on spaces and/or commas.
# Valid values for version_path_separator are:
#
# version_path_separator = :
# version_path_separator = ;
# version_path_separator = space
version_path_separator = os # Use os.pathsep. Default configuration used for new projects.
# the output encoding used when revision files
# are written from script.py.mako
# output_encoding = utf-8
# sqlalchemy.url = driver://user:pass@localhost/dbname
sqlalchemy.url = mysql+asyncmy://%(DB_USERNAME)s:%(DB_PASSWORD)s@%(DB_HOST)s:%(DB_PORT)s/%(DB_DATABASE)s
[post_write_hooks]
# post_write_hooks defines scripts or Python functions that are run
# on newly generated revision scripts. See the documentation for further
# detail and examples
# format using "black" - use the console_scripts runner, against the "black" entrypoint
hooks = black
black.type = console_scripts
black.entrypoint = black
black.options = -l 79 REVISION_SCRIPT_FILENAME
# Logging configuration
[loggers]
keys = root,sqlalchemy,alembic
[handlers]
keys = console
[formatters]
keys = generic
[logger_root]
level = WARN
handlers = console
qualname =
[logger_sqlalchemy]
level = WARN
handlers =
qualname = sqlalchemy.engine
[logger_alembic]
level = INFO
handlers =
qualname = alembic
[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic
[formatter_generic]
format = %(levelname)-5.5s [%(name)s] %(message)s
datefmt = %H:%M:%S

50
alembic/README.md Normal file
View File

@ -0,0 +1,50 @@
# alembic 目录
## 说明
该目录包含 [SQLAlchemy](https://www.sqlalchemy.org/) 数据库版本管理工具 [alembic](https://alembic.sqlalchemy.org/) 的迁移脚本migrations
这里数据库版本指的是本项目的数据库表结构,并不是 MySQL 版本。
## 常用功能
### 1. 升级版本
拉取代码以后,如果发现有新的 `alembic/versions/xxxxxxx.py`,执行命令进行数据库迁移。
``` shell
# 检查将要执行的 SQL
alembic upgrade head --sql
# 执行升级 SQL
alembic upgrade head
```
### 2. 创建一个迁移版本(自动)
新增或修改 Model 以后运行下面的命令alembic 将自动比较 model 定义和本地数据库的差异,然后创建一个新的迁移脚本
```python
# 举例:新增 Model
class User(SQLModel, table=True):
__table_args__ = dict(mysql_charset='utf8mb4', mysql_collate="utf8mb4_general_ci")
id: int = Field(primary_key=True)
name: str = Field()
```
``` shell
# 引号内是本次迁移的名字,类似 git commit message 请保持可读性
alembic revision --autogenerate -m "add_xxx_to_xxx_table"
```
创建以后,可以使用 `black` 或其他工具格式化迁移脚本,然后执行升级
### 3. 创建一个迁移版本(手动)
手动写迁移脚本,一般用于修改数据的情况,比如新增了一个字段,需要从其他表把数据读到新字段里。
``` shell
alembic revision -m "add_xxx_to_xxx_table"
```
> 通常情况下,如果上线后升级版本出错,不建议使用数据库降级命令,数据库降级难以管理,建议另外写一个迁移脚本进行手动降级

108
alembic/env.py Normal file
View File

@ -0,0 +1,108 @@
import asyncio
from logging.config import fileConfig
from sqlalchemy import engine_from_config
from sqlalchemy import pool
from sqlalchemy.engine import Connection
from sqlalchemy.ext.asyncio import AsyncEngine
from sqlmodel import SQLModel
from alembic import context
# this is the Alembic Config object, which provides
# access to the values within the .ini file in use.
config = context.config
# Interpret the config file for Python logging.
# This line sets up loggers basically.
if config.config_file_name is not None:
fileConfig(config.config_file_name)
# register our models for alembic to auto-generate migrations
from utils.manager import ModulesManager
manager = ModulesManager()
manager.refresh_list("core/*")
manager.refresh_list("jobs/*")
manager.refresh_list("plugins/genshin/*")
manager.refresh_list("plugins/system/*")
manager.import_module()
target_metadata = SQLModel.metadata
# other values from the config, defined by the needs of env.py,
# can be acquired:
# my_important_option = config.get_main_option("my_important_option")
# ... etc.
# here we allow ourselves to pass interpolation vars to alembic.ini
# from the application config module
from config import config as appConfig
section = config.config_ini_section
config.set_section_option(section, "DB_HOST", appConfig.mysql["host"])
config.set_section_option(section, "DB_PORT", str(appConfig.mysql["port"]))
config.set_section_option(section, "DB_USERNAME", appConfig.mysql["user"])
config.set_section_option(section, "DB_PASSWORD", appConfig.mysql["password"])
config.set_section_option(section, "DB_DATABASE", appConfig.mysql["database"])
def run_migrations_offline() -> None:
"""Run migrations in 'offline' mode.
This configures the context with just a URL
and not an Engine, though an Engine is acceptable
here as well. By skipping the Engine creation
we don't even need a DBAPI to be available.
Calls to context.execute() here emit the given string to the
script output.
"""
url = config.get_main_option("sqlalchemy.url")
context.configure(
url=url,
target_metadata=target_metadata,
literal_binds=True,
dialect_opts={"paramstyle": "named"},
)
with context.begin_transaction():
context.run_migrations()
def do_run_migrations(connection: Connection) -> None:
context.configure(connection=connection, target_metadata=target_metadata)
with context.begin_transaction():
context.run_migrations()
async def run_migrations_online() -> None:
"""Run migrations in 'online' mode.
In this scenario we need to create an Engine
and associate a connection with the context.
"""
connectable = AsyncEngine(
engine_from_config(
config.get_section(config.config_ini_section),
prefix="sqlalchemy.",
poolclass=pool.NullPool,
future=True,
)
)
async with connectable.connect() as connection:
await connection.run_sync(do_run_migrations)
await connectable.dispose()
if context.is_offline_mode():
run_migrations_offline()
else:
asyncio.run(run_migrations_online())

25
alembic/script.py.mako Normal file
View File

@ -0,0 +1,25 @@
"""${message}
Revision ID: ${up_revision}
Revises: ${down_revision | comma,n}
Create Date: ${create_date}
"""
from alembic import op
import sqlalchemy as sa
import sqlmodel
${imports if imports else ""}
# revision identifiers, used by Alembic.
revision = ${repr(up_revision)}
down_revision = ${repr(down_revision)}
branch_labels = ${repr(branch_labels)}
depends_on = ${repr(depends_on)}
def upgrade() -> None:
${upgrades if upgrades else "pass"}
def downgrade() -> None:
${downgrades if downgrades else "pass"}

View File

@ -0,0 +1,166 @@
"""init
Revision ID: 9e9a36470cd5
Revises:
Create Date: 2022-09-01 16:55:20.372560
"""
from alembic import op
import sqlalchemy as sa
import sqlmodel
# revision identifiers, used by Alembic.
revision = "9e9a36470cd5"
down_revision = None
branch_labels = None
depends_on = None
def upgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
op.create_table(
"question",
sa.Column("id", sa.Integer(), nullable=False),
sa.Column("text", sqlmodel.sql.sqltypes.AutoString(), nullable=True),
sa.PrimaryKeyConstraint("id"),
mysql_charset="utf8mb4",
mysql_collate="utf8mb4_general_ci",
)
op.create_table(
"user",
sa.Column(
"region",
sa.Enum("NULL", "HYPERION", "HOYOLAB", name="regionenum"),
nullable=True,
),
sa.Column("id", sa.Integer(), nullable=False),
sa.Column("user_id", sa.Integer(), nullable=False),
sa.Column("yuanshen_uid", sa.Integer(), nullable=True),
sa.Column("genshin_uid", sa.Integer(), nullable=True),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("user_id"),
mysql_charset="utf8mb4",
mysql_collate="utf8mb4_general_ci",
)
op.create_table(
"admin",
sa.Column("id", sa.Integer(), nullable=False),
sa.Column("user_id", sa.Integer(), nullable=False),
sa.ForeignKeyConstraint(
["user_id"],
["user.user_id"],
),
sa.PrimaryKeyConstraint("id"),
mysql_charset="utf8mb4",
mysql_collate="utf8mb4_general_ci",
)
op.create_table(
"answer",
sa.Column("question_id", sa.Integer(), nullable=True),
sa.Column("id", sa.Integer(), nullable=False),
sa.Column("is_correct", sa.Boolean(), nullable=True),
sa.Column("text", sqlmodel.sql.sqltypes.AutoString(), nullable=True),
sa.ForeignKeyConstraint(
["question_id"],
["question.id"],
onupdate="RESTRICT",
ondelete="RESTRICT",
),
sa.PrimaryKeyConstraint("id"),
mysql_charset="utf8mb4",
mysql_collate="utf8mb4_general_ci",
)
op.create_table(
"hoyoverse_cookies",
sa.Column("cookies", sa.JSON(), nullable=True),
sa.Column(
"status",
sa.Enum(
"STATUS_SUCCESS",
"INVALID_COOKIES",
"TOO_MANY_REQUESTS",
name="cookiesstatusenum",
),
nullable=True,
),
sa.Column("id", sa.Integer(), nullable=False),
sa.Column("user_id", sa.Integer(), nullable=True),
sa.ForeignKeyConstraint(
["user_id"],
["user.user_id"],
),
sa.PrimaryKeyConstraint("id"),
mysql_charset="utf8mb4",
mysql_collate="utf8mb4_general_ci",
)
op.create_table(
"mihoyo_cookies",
sa.Column("cookies", sa.JSON(), nullable=True),
sa.Column(
"status",
sa.Enum(
"STATUS_SUCCESS",
"INVALID_COOKIES",
"TOO_MANY_REQUESTS",
name="cookiesstatusenum",
),
nullable=True,
),
sa.Column("id", sa.Integer(), nullable=False),
sa.Column("user_id", sa.Integer(), nullable=True),
sa.ForeignKeyConstraint(
["user_id"],
["user.user_id"],
),
sa.PrimaryKeyConstraint("id"),
mysql_charset="utf8mb4",
mysql_collate="utf8mb4_general_ci",
)
op.create_table(
"sign",
sa.Column(
"time_created",
sa.DateTime(timezone=True),
server_default=sa.text("now()"),
nullable=True,
),
sa.Column("time_updated", sa.DateTime(timezone=True), nullable=True),
sa.Column(
"status",
sa.Enum(
"STATUS_SUCCESS",
"INVALID_COOKIES",
"ALREADY_CLAIMED",
"GENSHIN_EXCEPTION",
"TIMEOUT_ERROR",
"BAD_REQUEST",
"FORBIDDEN",
name="signstatusenum",
),
nullable=True,
),
sa.Column("id", sa.Integer(), nullable=False),
sa.Column("user_id", sa.Integer(), nullable=False),
sa.Column("chat_id", sa.Integer(), nullable=True),
sa.ForeignKeyConstraint(
["user_id"],
["user.user_id"],
),
sa.PrimaryKeyConstraint("id"),
mysql_charset="utf8mb4",
mysql_collate="utf8mb4_general_ci",
)
# ### end Alembic commands ###
def downgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
op.drop_table("sign")
op.drop_table("mihoyo_cookies")
op.drop_table("hoyoverse_cookies")
op.drop_table("answer")
op.drop_table("admin")
op.drop_table("user")
op.drop_table("question")
# ### end Alembic commands ###

View File

@ -1,8 +1,9 @@
from typing import Optional
from sqlmodel import SQLModel, Field
class Admin(SQLModel, table=True):
id: Optional[int] = Field(default=None, primary_key=True)
user_id: int = Field()
__table_args__ = dict(mysql_charset='utf8mb4', mysql_collate="utf8mb4_general_ci")
id: int = Field(primary_key=True)
user_id: int = Field(foreign_key="user.user_id")

View File

@ -11,8 +11,10 @@ class CookiesStatusEnum(int, enum.Enum):
class Cookies(SQLModel):
id: Optional[int] = Field(default=None, primary_key=True)
user_id: Optional[int] = Field()
__table_args__ = dict(mysql_charset='utf8mb4', mysql_collate="utf8mb4_general_ci")
id: int = Field(primary_key=True)
user_id: Optional[int] = Field(foreign_key="user.user_id")
cookies: Optional[Dict[str, str]] = Field(sa_column=Column(JSON))
status: Optional[CookiesStatusEnum] = Field(sa_column=Column(Enum(CookiesStatusEnum)))

View File

@ -1,6 +1,6 @@
from typing import List, Optional
from sqlmodel import SQLModel, Field
from sqlmodel import SQLModel, Field, Column, Integer, ForeignKey
from models.baseobject import BaseObject
from models.types import JSONDict
@ -8,15 +8,24 @@ from models.types import JSONDict
class AnswerDB(SQLModel, table=True):
__tablename__ = 'answer'
id: Optional[int] = Field(default=None, primary_key=True)
question_id: Optional[int] = Field()
__table_args__ = dict(mysql_charset='utf8mb4', mysql_collate="utf8mb4_general_ci")
id: int = Field(primary_key=True)
question_id: Optional[int] = Field(
sa_column=Column(
Integer,
ForeignKey("question.id", ondelete="RESTRICT", onupdate="RESTRICT")
)
)
is_correct: Optional[bool] = Field()
text: Optional[str] = Field()
class QuestionDB(SQLModel, table=True):
__tablename__ = 'question'
id: Optional[int] = Field(default=None, primary_key=True)
__table_args__ = dict(mysql_charset='utf8mb4', mysql_collate="utf8mb4_general_ci")
id: int = Field(primary_key=True)
text: Optional[str] = Field()

View File

@ -18,9 +18,11 @@ class SignStatusEnum(int, enum.Enum):
class Sign(SQLModel, table=True):
id: Optional[int] = Field(default=None, primary_key=True)
user_id: int = Field()
chat_id: int = Field()
__table_args__ = dict(mysql_charset='utf8mb4', mysql_collate="utf8mb4_general_ci")
id: int = Field(primary_key=True)
user_id: int = Field(foreign_key="user.user_id")
chat_id: Optional[int] = Field(default=None)
time_created: Optional[datetime] = Field(sa_column=Column(DateTime(timezone=True), server_default=func.now()))
time_updated: Optional[datetime] = Field(sa_column=Column(DateTime(timezone=True), onupdate=func.now()))
status: Optional[SignStatusEnum] = Field(sa_column=Column(Enum(SignStatusEnum)))

View File

@ -6,8 +6,10 @@ from models.base import RegionEnum
class User(SQLModel, table=True):
id: Optional[int] = Field(default=None, primary_key=True)
user_id: int = Field()
yuanshen_uid: int = Field()
genshin_uid: int = Field()
__table_args__ = dict(mysql_charset='utf8mb4', mysql_collate="utf8mb4_general_ci")
id: int = Field(primary_key=True)
user_id: int = Field(unique=True)
yuanshen_uid: Optional[int] = Field()
genshin_uid: Optional[int] = Field()
region: Optional[RegionEnum] = Field(sa_column=Column(Enum(RegionEnum)))

1903
poetry.lock generated Normal file

File diff suppressed because it is too large Load Diff

View File

@ -24,6 +24,8 @@ pyppeteer = "^1.0.2"
aiofiles = "^0.8.0"
python-dotenv = "^0.20.0"
PyMySQL = "^1.0.2"
alembic = "^1.8.1"
black = "^22.8.0"
[build-system]