Analysis of ibis issues.

This document explores the project management of issues in the ibis project. It produces a csv that can be used in "annual_tags.csv".

import ibis, pandas; from poser import *

Cache the results after we grab them to improve reuse. Delete the cache file to reset cache.

λ['requests_cache.install_cache']('issues')

We'll only concern ourselves with some backends specifically.

backends =\

omnisci spark postgres bigquery pandas sqlite impala kudu geospatial clickhouse mysql sqlalchemy

get_issues: (

Paginated request for the ibis issues.

) = (
    λ
    ['https://api.github.com/repos/ibis-project/ibis/issues?page={}&state=all'.format]
    .partial('requests.get', params=dict(access_token=__import__('os').environ['ACCESS_TOKEN']))
    [Λ.json()]
)

i, last = 0, range(30)

Iterate through the issues. The Github API returns 30 results if there are more results on another page. This look is loading the list of issues and caching them.

while len(last) == 30:
    i +=1
    last = get_issues(i)

Tidy the issues into a dataframe.

issues = λ.range(1,i).map(λ+get_issues+pandas.DataFrame)[pandas.concat]().set_index('id')
times = λ(issues.columns) / λ.endswith('_at') + list + ...
issues[times] = issues[times].apply(pandas.to_datetime)
del times

We accessed 2190 from the ibis project.

Extract the label_names from the issues.

label_names = issues['labels'].apply(λ['pandas.Series']).stack().apply(λ['pandas.Series']).name
label_names = label_names.pipe(λ['pandas.get_dummies']).pipe(lambda df: df.groupby(df.index.get_level_values(0)).sum())

Join the labels with the original dataframe and compose an annual analysis of the backends.

issues_tags = label_names.join(issues).set_index('created_at').groupby(pandas.Grouper(freq='1Y')).sum()[backends.split()]
issues_tags = issues_tags.set_index(pandas.Index(issues_tags.index.year, name='year'))
issues_tags = issues_tags.replace({0:''})

issues_tags.to_csv('annual_tags.csv')