Extend data

Data structure

zvt use sql database(default sqlite, and easy extend to others) to store data and provide uniform api to record and query data.(Thanks to the great Sqlalchemy)

As we know, TradableEntity and ActorEntity is everything about the market, data is the events happen on them. The Schema of the data would always in following structure:


It shows how we define the Entity in the market:

  • entity_type

tradable entity type, e.g. stock, future, bond and so on.

  • exchange

the exchange of entity

  • code

the code of the entity

Naturally the id of the entity is defined:


And the common fields:

  • entity_id

entity id

  • timestamp

happen time

  • id

the id of the record, most of time, it’s:


Data definition

zvt data is defined by three concepts:

  • Database

The rule is one module define one db.

  • Schema

The Schema defined in the module belongs to the Database defined inside.

  • Provider

The Database could have different providers

Many pre supported data is in zvt.domain modules, e.g. Finance data, you could refer them when you want to extend data.

Key steps to add data

Let’s show the key steps to add data.

1. Intent

I want to add some data about the news on Stock.

2. Find data source

The data source could be whatever you can see in the web ——— just write some web crawlers to get them, e.g, most of pre-implemented recorders. And you could use data from commercial data provider in the same way, e.g. joinquant recorders.

Let’s show the crawler way here:

Open eastmoney wap and Find news api

The news item result:

  "Art_ShowTime": "2022-02-09 14:12:30",
  "Art_Image": "",
  "Art_MediaName": "每日经济新闻",
  "Art_Code": "202202092271136373",
  "Art_Title": "首创证券维持索菲亚买入评级 公司简评报告:多重因素影响短期压制业绩 看好2022年利润修复",
  "Art_SortStart": "1644387150036373",
  "Art_VideoCount": 0,
  "Art_OriginUrl": "http://finance.eastmoney.com/news/1354,202202092271136373.html",
  "Art_Url": "http://finance.eastmoney.com/a/202202092271136373.html"

3. Define Schema

No matter what the format of the data outside zvt is, we use zvt simple and uniform contract to define them.

>>> from sqlalchemy import Column, String
>>> from sqlalchemy.orm import declarative_base
>>> from zvt.contract import Mixin
>>> from zvt.contract.register import register_schema
>>> NewsBase = declarative_base()
>>> class StockNews(NewsBase, Mixin):
>>>     __tablename__ = "stock_news"
>>>     #: news title
>>>     news_title = Column(String)
>>> register_schema(providers=["em"], db_name="stock_news", schema_base=NewsBase, entity_type="stock")

Check the defined db

>>> StockNews.get_storages()


As you see, the db file name format is:


If you have another provider, e.g. sina, just register it too:

>>> register_schema(providers=["em", "sina"], db_name="stock_news", schema_base=NewsBase, entity_type="stock")

And you could find another db file:

>>> StockNews.get_storages()


The advantage of this mechanism is:

  • schema is the way you want

  • provider could be switched seamlessly

4. Implement recorder

Let’s implement a recorder for StockNews.

import pandas as pd
from zvt.contract.api import df_to_db
from zvt.contract.recorder import FixedCycleDataRecorder
from zvt.domain import Stock
from zvt.domain.misc.stock_news import StockNews
from zvt.recorders.em import em_api

class EMStockNewsRecorder(FixedCycleDataRecorder):
    original_page_url = "https://wap.eastmoney.com/quote/stock/0.002572.html"
    url = "https://np-listapi.eastmoney.com/comm/wap/getListInfo?cb=callback&client=wap&type=1&mTypeAndCode=0.002572&pageSize=200&pageIndex={}&callback=jQuery1830017478247906740352_1644568731256&_=1644568879493"

    # the entity class you record for
    entity_schema = Stock
    # connect schema with recorder
    data_schema = StockNews
    # entity data provider
    entity_provider = "em"
    # data schema provider
    provider = "em"

    def record(self, entity, start, end, size, timestamps):
        news = em_api.get_news(entity_id=entity.id)
        df = pd.DataFrame.from_records(news)
        df_to_db(df=df, data_schema=self.data_schema, provider=self.provider, force_update=self.force_update)

After that, you could use the data in zvt way:

>>> StockNews.record_data(provider="em", entity_id="stock_sz_002572")
>>> df = StockNews.query_data(entity_id="stock_sz_002572")
>>> print(df)

                                          id        entity_id           timestamp                                     news_title
0    stock_sz_002572_2019-05-20 19:50:42  stock_sz_002572 2019-05-20 19:50:42                索菲亚(002572.SZ):股价回撤超65% 是低吸机会吗?
1    stock_sz_002572_2019-05-24 22:39:47  stock_sz_002572 2019-05-24 22:39:47                             索菲亚拟发行不超过5亿元超短期融资券
2    stock_sz_002572_2019-05-24 22:48:54  stock_sz_002572 2019-05-24 22:48:54              索菲亚(002572.SZ)拟终止发行不超10亿元的可转换公司债券
3    stock_sz_002572_2019-06-04 14:06:42  stock_sz_002572 2019-06-04 14:06:42              索菲亚(002572.SZ)截至5月底已累计回购2.02亿元的股份
4    stock_sz_002572_2019-06-13 06:24:44  stock_sz_002572 2019-06-13 06:24:44                                索菲亚功臣王飚能否扶起汉森中国
..                                   ...              ...                 ...                                            ...
367  stock_sz_002572_2022-02-07 14:04:40  stock_sz_002572 2022-02-07 14:04:40         开源证券维持索菲亚买入评级 近期获6份券商研报关注 目标均价涨幅59.82%
368  stock_sz_002572_2022-02-07 15:32:46  stock_sz_002572 2022-02-07 15:32:46                             【调研快报】索菲亚接待机构投资者调研
369  stock_sz_002572_2022-02-08 16:50:37  stock_sz_002572 2022-02-08 16:50:37                             索菲亚:公司承接了容东片区安置房项目
370  stock_sz_002572_2022-02-08 21:59:00  stock_sz_002572 2022-02-08 21:59:00       9亿坏账拖累净利下降九成 家具巨头索菲亚“甩包袱”起跑 腰斩的股价能否趁势抬头?
371  stock_sz_002572_2022-02-09 14:12:30  stock_sz_002572 2022-02-09 14:12:30  首创证券维持索菲亚买入评级 公司简评报告:多重因素影响短期压制业绩 看好2022年利润修复

[372 rows x 4 columns]

Add new TradableEntity

It’s human nature to like the new and hate the old. Adding new TradableEntity is easy in zvt.

And from a higher perspective, trading is everywhere. you make trading everytime when you make the decision.

So you could treat Country as TradableEntity and make trading when making decision where to live or invest.

Let’s show the key steps to add new TradableEntity.

1. Define entity Schema

# -*- coding: utf-8 -*-

from sqlalchemy import Column, String, Float
from sqlalchemy.orm import declarative_base

from zvt.contract.schema import TradableEntity
from zvt.contract.register import register_schema, register_entity

CountryMetaBase = declarative_base()

class Country(CountryMetaBase, TradableEntity):
    __tablename__ = "country"

    #: 区域
    #: region
    region = Column(String(length=128))
    #: 首都
    #: capital city
    capital_city = Column(String(length=128))
    #: 收入水平
    #: income level
    income_level = Column(String(length=64))
    #: 贷款类型
    #: lending type
    lending_type = Column(String(length=64))
    #: 经度
    #: longitude
    longitude = Column(Float)
    #: 纬度
    #: latitude
    latitude = Column(Float)

register_schema(providers=["wb"], db_name="country_meta", schema_base=CountryMetaBase)

entity_type, exchange and code define the entity, for country, it’s in following way:

entity_type = "country"
exchange = "galaxy"
code = "iso code"

e.g. country_galaxy_CN = China, country_galaxy_US = United States of America

2. Implement recorder for the entity

from zvt.contract.api import df_to_db
from zvt.contract.recorder import Recorder
from zvt.domain.meta.country_meta import Country
from zvt.recorders.wb import wb_api

class WBCountryRecorder(Recorder):
    provider = "wb"
    data_schema = Country

    def run(self):
        df = wb_api.get_countries()
        df_to_db(df=df, data_schema=self.data_schema, provider=self.provider, force_update=self.force_update)

3. Define schema for the entity

e.g treasury yield of the country

# -*- coding: utf-8 -*-
from sqlalchemy import Column, String, Float
from sqlalchemy.orm import declarative_base

from zvt.contract import Mixin
from zvt.contract.register import register_schema

CurrencyBase = declarative_base()

class TreasuryYield(CurrencyBase, Mixin):
    __tablename__ = "treasury_yield"

    code = Column(String(length=32))

    # 2年期
    yield_2 = Column(Float)
    # 5年期
    yield_5 = Column(Float)
    # 10年期
    yield_10 = Column(Float)
    # 30年期
    yield_30 = Column(Float)

register_schema(providers=["em"], db_name="currency", schema_base=CurrencyBase)

And the recorder for the schema

4. Use them in zvt way

Find the rich country:

>>> from zvt.domain import Country
>>> Country.record_data()
>>> df = Country.query_data()
>>> df[df['income_level']=='High income']

                   id          entity_id timestamp entity_type exchange code                    name list_date end_date                      region      capital_city income_level    lending_type   longitude   latitude
0    country_galaxy_AW  country_galaxy_AW      None     country   galaxy   AW                   Aruba      None     None  Latin America & Caribbean         Oranjestad  High income  Not classified  -70.016700  12.516700
7    country_galaxy_AD  country_galaxy_AD      None     country   galaxy   AD                 Andorra      None     None       Europe & Central Asia  Andorra la Vella  High income  Not classified    1.521800  42.507500
9    country_galaxy_AE  country_galaxy_AE      None     country   galaxy   AE    United Arab Emirates      None     None  Middle East & North Africa         Abu Dhabi  High income  Not classified   54.370500  24.476400
13   country_galaxy_AG  country_galaxy_AG      None     country   galaxy   AG     Antigua and Barbuda      None     None  Latin America & Caribbean       Saint John's  High income            IBRD  -61.845600  17.117500
14   country_galaxy_AU  country_galaxy_AU      None     country   galaxy   AU               Australia      None     None         East Asia & Pacific          Canberra  High income  Not classified  149.129000 -35.282000
..                 ...                ...       ...         ...      ...  ...                     ...       ...      ...                         ...               ...          ...             ...         ...        ...
277  country_galaxy_TW  country_galaxy_TW      None     country   galaxy   TW           Taiwan, China      None     None         East Asia & Pacific                    High income  Not classified         NaN        NaN
282  country_galaxy_UY  country_galaxy_UY      None     country   galaxy   UY                 Uruguay      None     None  Latin America & Caribbean         Montevideo  High income            IBRD  -56.067500 -34.894100
283  country_galaxy_US  country_galaxy_US      None     country   galaxy   US           United States      None     None               North America   Washington D.C.  High income  Not classified  -77.032000  38.889500
287  country_galaxy_VG  country_galaxy_VG      None     country   galaxy   VG  British Virgin Islands      None     None  Latin America & Caribbean          Road Town  High income  Not classified  -64.623056  18.431389
288  country_galaxy_VI  country_galaxy_VI      None     country   galaxy   VI   Virgin Islands (U.S.)      None     None  Latin America & Caribbean   Charlotte Amalie  High income  Not classified  -64.896300  18.335800

[80 rows x 15 columns]

Compare treasury yields of different maturities:

>>> from zvt.domain import TreasuryYield
>>> from zvt.api.intent import compare
>>> TreasuryYield.record_data()
>>> compare(codes=["US"], schema=TreasuryYield, columns=["yield_2", "yield_10", "yield_30"])