linearmodels
The linearmodels pack has 25 rules, and xpd can fix 14 of them.
The linearmodels rules report formulas and arguments that fit another model than the code reads
as, covariance options that give other standard errors than the code asks for, code that raises on
linearmodels 7.1, statsmodels names read on linearmodels results, and forms that a release removed
or deprecated. They run in each file that imports linearmodels.
linearmodels fits panel models, instrument models,
systems of equations and factor models of asset prices.
Every number about the linearmodels rules was measured on linearmodels 7.1 with pandas 3.1.1,
NumPy 2.5.5, SciPy 1.18.1, statsmodels 0.15.0 and formulaic 1.2.2 on Python 3.13, and a number for another
release names it. The data sets wage_panel, mroz and french are the ones in
linearmodels.datasets. The examples assume that the linearmodels names they use are imported and
that sm is statsmodels.api. Names such as data, y, X and instruments stand for values
from your program.
Most of the pack reports code written from memory of statsmodels, because linearmodels looks like statsmodels and differs from it in the following small ways:
| What | statsmodels | linearmodels |
|---|---|---|
| intercept in a formula | added | not added |
| covariance names | "nonrobust", "HC1", "cluster" |
"unadjusted", "robust", "clustered" |
| result names | bse, tvalues, resid |
std_errors, tstats, resids |
IV2SLS arguments |
(endog, exog, instrument=None), in statsmodels.sandbox.regression.gmm |
(dependent, exog, endog, instruments) |
summary |
a method | a property |
OLS, IV2SLS, IVGMM and SUR exist in statsmodels as well. A rule about one of the four
reports a call only with proof that the class comes from linearmodels. The proof is the module in
front of the name, as in linearmodels.IV2SLS or lm.IV2SLS after import linearmodels as lm, or
an import of the name from linearmodels in the same file. A file that imports the same name from
both libraries gives no proof, so the rules about the four classes stay silent there. The other 15
model classes of linearmodels.__all__ have names of their own and need no proof.
A rule about fit or about a result reads only an object it can follow to a linearmodels class.
Such an object is a call on the class itself, such as PanelOLS(y, X).fit(...), or a name that
its function, or the module outside any function, assigns from such calls and from nothing else.
The rule leaves alone a name that is also assigned from anything else, that a loop, a with
statement or a lambda parameter binds, or that comes in as an argument. A statsmodels model in the
same file is therefore not reported. The 950 public files hold 585 cov_type= strings with a
statsmodels name, such as "HC3" and "cluster", and no rule reports one of them.
A rule reads an argument only when it is written out. A cov_type, a formula or a flag that is a
name, an f-string or a value unpacked from a dictionary is left alone.
A rule document whose advice needs a newer release carries since. xpd reads the installed version
from the project's venv and skips the document on an older release. When the venv has no
linearmodels, or there is no venv, every document runs. Each since was checked by installing
from PyPI the release before it and the release at it, and running the form on both:
since |
Releases run |
|---|---|
| 4.15 | 4.14 and 4.15 |
| 4.18 | 4.17 and 4.18 |
| 5.0 | 4.31 and 5.0 |
| 7.1 | 7.0 and 7.1 |
The proof each rule needs
Section titled “The proof each rule needs”Each error and warning document ships only with a proof that was run, and the message quotes
what the proof measured. A proof is one of the following kinds:
| Proof | What runs | When it passes |
|---|---|---|
| result | The old and the new form on the same data, with what each returns printed. | The old form gives another number than the code reads as, with no error and no warning, such as a coefficient of 0.2184 where 0.0338 is the one with an intercept. |
| interpreter | The old form under python -W error. |
The old form raises an exception or a warning, and the new form runs clean. Old releases warn about their own dependencies, so on the old releases only a warning from linearmodels counted. |
| simulation | At least 1,000 draws of data whose true coefficient is known. | It counts how often the 95% interval covers the coefficient, or how often a true null is rejected at 5%. |
| speed | Both forms on one thread, as the median of 5 fits, at two sizes. | The new form is at least 1.5 times faster at both. |
A document is info when its advice depends on something the code doesn't show, such as the dtype
of a column or how the errors are related. A form that was run on 7.1 and on 7.0 gave the same
result on both unless the text says otherwise.
The rules were run over two sets of code:
- the library's own code at 7.1, 258 files in all:
- the 229 Python files of the repository, of which 165 are tests
- the code cells of its 12 notebooks
- the code blocks of 9
.rstand.mdpages - the doctest lines of 8 modules
- 950 files from 623 public repositories that mention
linearmodels, found through the GitHub code search, at most 4 from one repository, with notebooks turned into their code cells
After the runs, the rules below were narrowed, and the table gives the reports that are left:
time-before-entity-indexreported 2 frames that no panel model reads, and one frame indexed by date and hour on purpose. The frame must now reach a panel model, andhouris a time.statsmodels-result-namereportedres.tvaluesbehindhasattr(res, "tstats"). Every rule about a form that raises now leaves code under ahasattrcheck alone.hand-written-hausmanreported a test of OLS against instruments, which the method of 7.1 doesn't do. Its message now nameswu_hausman()as well.absorb-not-categoricalmatched its own name in a comment. It now looks forcategory.
| Rule | Severity | Proof | Reports on the library | Reports on 950 public files | Fix |
|---|---|---|---|---|---|
formula-without-constant |
warning | result: 0.2184 against 0.0338 | 3 | 13 | none |
effects-outside-panel-ols |
error | result: equal coefficients with and without the term | 0 | 0 | none |
effects-keyword-with-formula |
error | interpreter: TypeError | 0 | 3 | none |
formula-power-term |
error | result: 2 columns where 3 were written | 1 | 1 | none |
first-difference-constant |
error | interpreter: ValueError | 0 | 0 | none |
weights-column-name |
error | interpreter: TypeError | 0 | 0 | none |
panel-ols-without-effects |
warning | result: 0.0349 against 0.0698 | 42 | 37 | none |
too-many-effects |
error | interpreter: ValueError | 0 | 0 | none |
time-before-entity-index |
warning | result: 0.1622 against 0.2483 | 1 | 1 | unsafe |
use-lsdv |
warning | speed: 17 times at the lowest | 2 | 2 | none |
clustered-without-clusters |
error | result: 0.0413 against 0.0494. Interpreter: TypeError | 4 | 31 | none |
cov-option-without-cov-type |
error | interpreter: ValueError, TypeError. Result: 0.2037 against 0.3002 | 0 | 0 | none |
unknown-cov-type |
error | interpreter: UnknownOptionError | 2 | 0 | none |
kernel-default-bandwidth |
warning | result: 0.8022 on 7.0 against 0.3794 on 7.1 | 22 | 1 | none |
panel-default-covariance |
info | simulation: 17.3% against 5.7% rejections | 325 | 372 | none |
iv-missing-arguments |
error | interpreter: TypeError | 0 | 2 | none |
iv-arguments-swapped |
warning | result: 0.2021 against 0.0664 | 0 | 0 | none |
iv-without-exog |
warning | simulation: 2.002 against 0.990 for a true 1.0 | 4 | 2 | none |
manual-two-stage |
warning | simulation: 99.6% against 95.7% coverage | 0 | 0 | none |
absorb-not-categorical |
info | result: 3.397 against 1.022 for a true 1.0 | 0 | 5 | none |
statsmodels-result-name |
error | interpreter: TypeError, AttributeError | 0 | 15 | unsafe |
invalid-compare-precision |
error | interpreter: ValueError | 0 | 0 | none |
hand-written-hausman |
info | none: the method runs on 7.1 and is absent on 7.0 | 4 | 25 | none |
removed-in-linearmodels |
error | interpreter: ModuleNotFoundError, AttributeError | 0 | 0 | unsafe |
deprecated-in-linearmodels |
warning | interpreter: FutureWarning. Result: equal order | 12 | 0 | safe |
A result proof gives the old form's number against the new form's number.
Of the 422 reports on the library, 399 are in its tests, which call the forms on purpose. They are
308 fits with the default covariance, 42 PanelOLS models without effects that are compared with
PooledOLS, 22 kernel estimators with the default bandwidth, and 11 uses of the deprecated names
inside the tests of the deprecations. One report names a line where the advice is wrong. The line is on a
documentation page that shows set_index(['year','firm']) for the PanelOLS that pandas had in
2017, which took the time first. One report is a defect of the library's own README, whose first
example writes exper + exper ** 2 and fits a model without the squared term.
Of the 510 reports on the 950 public files, 108 are errors or warnings, and each was read. None names
a line where the rule's claim is false. The other 402 are info. Of them, 20 of the 372
panel-default-covariance reports, 14 hand-written-hausman reports and all 5
absorb-not-categorical reports were read, and each is a fit or a model of the kind the rule
describes. unknown-cov-type reports none of the 585 cov_type= strings with a statsmodels name,
and the 20 of them that were read are calls on statsmodels models.
Recall was not measured, so nothing here says how often a rule misses the form it describes.
Formulas
Section titled “Formulas”Each rule in the group reports a formula that fits another model than it reads as. linearmodels
parses formulas with formulaic, and every model has a from_formula. The rules read a formula only
when it is a string in the from_formula call, as the first argument, the formula keyword, or a
value of the dictionary that a system model takes.
| Rule | Severity | Fix | What it finds |
|---|---|---|---|
| formula-without-constant | warning | none | linearmodels adds no intercept to a formula, where statsmodels and R add one, so the estimates change. |
| effects-outside-panel-ols | error | none | Only PanelOLS reads effect terms in a formula, and the other panel models drop them without an error. |
| effects-keyword-with-formula | error | unsafe | PanelOLS.from_formula has no entity_effects argument and raises TypeError, because the formula holds the effects. |
| formula-power-term | error | unsafe | A formula reads ** as a power of interactions, so exper**2 is just exper, with no error. |
| first-difference-constant | error | unsafe | FirstDifferenceOLS takes no constant, so a 1 term or add_constant raises ValueError. |
| weights-column-name | error | unsafe | weights takes the values and not the name of a column, so a name raises. |
Panel models
Section titled “Panel models”Each rule in the group reports a panel model that reads its index or its effects in another way
than the code means. A panel model takes a frame whose index has two levels, the entity first and
the time second. PanelOLS is the only class with effects, and it has none until it is told to.
| Rule | Severity | Fix | What it finds |
|---|---|---|---|
| panel-ols-without-effects | warning | none | PanelOLS without effects fits pooled OLS, despite its name. |
| too-many-effects | error | none | PanelOLS takes at most two effects, so three raise ValueError. |
| time-before-entity-index | warning | unsafe | The panel models read the first index level as the entity and the second as the time, and here the time comes first, so the effects are on the wrong level. |
| use-lsdv | warning | unsafe | use_lsdv=True builds a dummy column for each entity, which was up to 1,508 times slower and gave the same coefficients. |
Covariance
Section titled “Covariance”Each rule in the group reports a covariance choice that gives other standard errors than the code
asks for, or that raises. fit takes cov_type and, next to it as keywords, the options of the
chosen covariance. Each model has its own short list of names, and none of them is a statsmodels
name.
| Rule | Severity | Fix | What it finds |
|---|---|---|---|
| clustered-without-clusters | error | none | cov_type="clustered" with no cluster variable puts each observation in a cluster of its own, which is the robust covariance under another name. |
| cov-option-without-cov-type | error | unsafe | A cluster or kernel option in a fit call without the matching cov_type raises an error instead of clustering. |
| unknown-cov-type | error | unsafe | The model does not take this cov_type, such as the statsmodels names "HC1" or "cluster", so fit raises. |
| kernel-default-bandwidth | warning | unsafe | The default kernel bandwidth changed in linearmodels 7.1, so results change between versions and 7.1 warns. |
| panel-default-covariance | info | none | fit without cov_type assumes the errors of one entity are unrelated over time, so it rejects true hypotheses far too often when they are related. |
Instruments and absorbed effects
Section titled “Instruments and absorbed effects”Each rule in the group reports an instrument model or an absorbed effect that is set up in another
way than the code means. IV2SLS, IVLIML, IVGMM, IVGMMCUE and IVJIVE take four arrays in
the order dependent, exog, endog, instruments. None of the four has a default, and none of the
models adds a constant.
| Rule | Severity | Fix | What it finds |
|---|---|---|---|
| iv-missing-arguments | error | none | The linearmodels instrument models take dependent, exog, endog and instruments in that order, so a call with fewer arguments raises TypeError. |
| iv-arguments-swapped | warning | unsafe | The instrument models take exog before endog, and these arguments are named the other way round, so the model instruments the wrong variables. |
| iv-without-exog | warning | none | With exog=None, the instrument model has no intercept, because linearmodels adds none, so the estimates are biased. |
| manual-two-stage | warning | none | Fitting a second regression on the fitted values of a first one is two stage least squares by hand, and its standard errors come from the wrong residuals. |
| absorb-not-categorical | info | none | AbsorbingLS absorbs a column as fixed effects only when its dtype is categorical, so a numeric id becomes one continuous regressor. |
Results
Section titled “Results”Read a linearmodels result by its own attribute names, pass compare a precision that it takes, and use the built-in Hausman test in place of one written by hand.
| Rule | Severity | Fix | What it finds |
|---|---|---|---|
| statsmodels-result-name | error | unsafe | A linearmodels result does not have the statsmodels names, so summary(), bse and similar names raise. |
| invalid-compare-precision | error | unsafe | compare takes only "tstats", "std_errors" or "pvalues" for precision, so another value raises ValueError. |
| hand-written-hausman | info | none | linearmodels has both Hausman tests built in, so a hand-written one is extra code to keep right. |
Removed and deprecated forms
Section titled “Removed and deprecated forms”Each rule in the group reports a form that a release removed or deprecated. Each form was run on the release before and the release at the version the rule names, both installed from PyPI.
| Rule | Severity | Fix | What it finds |
|---|---|---|---|
| removed-in-linearmodels | error | unsafe | These modules and methods were removed from linearmodels, so the import or the call raises. |
| deprecated-in-linearmodels | warning | unsafe | These imports, keywords and names warn on linearmodels 7.1, and 8.0 removes them. |
What the pack leaves out
Section titled “What the pack leaves out”The pack has no rule for the following forms:
- Forms that a pattern can't tell apart from right code:
- Inputs whose labels don't match, which raise
ValueErroron 7.1 where 7.0 paired them by position. A pattern can't see an index. - A frame without a two level index, a time level of text, and regressors that the effects absorb. Each raises with a clear message, and none shows in the code.
- The changes of 7.1 to weighted models, to
FamaMacBethp-values and to the Hausman and specification tests. The code is the same on both releases, so a rule would report every use.
- Inputs whose labels don't match, which raise
- Forms that were tried and gave no reason for a rule:
check_rank=False, because on columns that are exactly collinear 7.1 raisedLinAlgError: Singular matrix. Where it was tried, the flag didn't give wrong numbers in silence.- Patsy functions that formulaic lacks.
te()raised, andcr()andcc()raised a rank error next to an intercept.Q(),center(),standardize(),bs()andC(x, Treatment(reference=...))all ran. No use of the three was found in the public files. - xxhash, which
AbsorbingLSuses for its cache when it is installed, as a package check. A fit with two absorbed columns took 0.134 s with it and 0.134 s without on 100,000 rows, and 1.130 s against 1.123 s on 1,000,000, with the default number of threads.
SUR,IV3SLS,IVSystemGMMand the factor models, which have no rule of their own. The formula, covariance and result rules read them.