Chase the Devil

The Role of Erfcx and Erfc in the Black-Scholes Formula

Sat, 30 May 2026 03:43:00 +0000

This note compares special-function choices at the level that matters to the implied volatility solvers: the normalized Black beta price, not standalone erfcx(x).

For x = log(F/K) <= 0 and total volatility s = sigma sqrt(T), the beta-space OTM call price is

B(x, s) = 0.5 * (exp(x/2) * erfc(q1) - exp(-x/2) * erfc(q2))
q1 = -(x/s + s/2) / sqrt(2)
q2 = -(x/s - s/2) / sqrt(2)

The corresponding forward-normalized OTM-call price is c = B(x, s) / sqrt(ex), with ex = exp(x).

Which erfcx?

Sun, 24 May 2026 03:43:00 +0000

The use of erfcx instead of direct erfc or CDF in a Black-Scholes implied volatility solver leads to gain in accuracy and performance in general. But which erfcx should we use?

This note compares practical erfcx implementations for Rust implied volatility solvers:

Commons: the local Rust port of Apache Commons Numbers BoostErf.erfcx
Cody: the Cody rational approximation exposed by the jaeckel = 0.2.0 crate during this comparison, originally coming from Netlib SLATEC library.
Johnson: Steven G. Johnson’s Faddeeva implementation, as exposed by errorfunctions = 0.2.0, faddeeva-sys = 0.1.0, and Julia SpecialFunctions.erfcx(::Float64)

The Commons-style implementation is a local Rust port of the Apache Commons Numbers/BoostErf approach. It computes the small central region through the local erf rational approximation and uses exp_m1(x*x) to avoid cancellation in exp(x^2) * erfc(x) near zero. For positive inputs it uses several rational subintervals: a central polynomial/rational erf branch, erfc-style rational branches around 0.5..1.5, 1.5..2.5, and 2.5..4, and a rational function taken from Rational Chebyshev approximations for the error function by W. J. Cody, Math. Comp., 1969, PP. 631-638 beyond that. For very large positive x, it returns 1 / (sqrt(pi) * x). For negative x, it uses the reflected form with a split-square exponential, returns Inf below the finite overflow cutoff, and otherwise computes 2 exp(x^2) - erfcx(|x|) in a way that preserves more bits near the overflow boundary.

A Faster Monotone Implied Volatiltty Solver

Sat, 23 May 2026 03:43:00 +0000

Choi, Huh and Su have a very good paper entitled Tighter uniform bounds for Black–Scholes implied volatility and the applications to root-finding. What’s particularly great is that it gives both a decent lower bound and a proof a monotone convergence using Newton’s method starting from this lower bound.

The industry standard for solving the Black-Scholes implied volatility is Peter Jäckel Let’s be rational. I was wondering if one could not create a robust mathematically backed solver at least as fast and accurate as Jäckel’s algorithm.

Almost Explicit Implied Volatility

Thu, 30 Apr 2026 18:10:00 +0000

Several years ago, I had explored accuracy and performance of different ways to imply the Black-Scholes volatility. Jherek Healy proposed some improvements over my naive algorithm on his blog. Recently, a Linkedin post mentioned a new paper from Wolfgang Schadner which presents an almost explicit formula for the implied volatility. Almost because it actually relies on some implementation of the quantile function for the inverse Gaussian distribution: it requires a specialized numerical algorithm in practice, such as the one from Giner and Smyth.

Owen Scrambling a la Burley

Thu, 08 Jan 2026 17:27:00 +0000

In my last post, I had a look at Quantlib implementation of a new scrambling method for Sobol due to Brent Burley of Walt Disney Studios Practical Hash-based Owen Scrambling.

Because it originates from the CG community, I had assumed that this was faster than the more classic scrambling ACM Algorithm 823 by Hickernell and Hong. I was wrong. It may be faster for specific use cases, but in general it is (much) slower, especially for the kind of (quasi) Monte-Carlo simulation that we use in quantitative finance.

Jack Audio in Opensuse Tumbleweed

Mon, 22 Dec 2025 17:27:00 +0000

I struggled a bit having Jack Audio Connection Kit working in Opensuse Tumbleweed. My error was to install the jack package. The solution is actually extremely simple: use pipewire-jack instead of jack.

sudo zypper in pipewire-jack qjackctl

Owen Scrambling in Quantlib

Wed, 17 Dec 2025 17:27:00 +0000

The state of the art of Sobol scrambling has changed slightly recently, thanks to the paper from Brent Burley of Walt Disney Studios Practical Hash-based Owen Scrambling. Before that, ACM Algorithm 823 by Hickernell and Hong was the usual reference. Brent Burley’s algorithm is supposedly both faster and with better properties. In particular, it performs both shuffling and scrambling.

Peter Caspers implemented the new algorithm in Quantlib. It’s neat to have some decent implementation in CPP. In applying the algorithm of the paper, a few sub-optimal choices were made:

Deep Neural Networks and Julia

Tue, 07 Oct 2025 17:27:00 +0000

Recently, I have spent some time on simple neural networks. The idea is to employ them as universal function approximators for some problems appearing in quantitative finance. There are some great papers on it such as the one from Liu et al. (2019) or Horvath et al. (2019) Deep Learning Volatility or Rosenbaum & Zhang (2021). Incidentally, I met Liu back when I was finishing my PhD in TU Delft around 2020.

I thought I would try out what Julia offers in terms of library for neural networks. Being a very trendy subject, and Julia a modern language for the scientific community, I had imagined the libraris to be of good quality (like the many I have been using in the past). Surprisingly, I was wrong.

Stochastic Collocation - Old And New

Tue, 05 Aug 2025 17:27:00 +0000

Thomas Roos recently put a preprint on SSRN called Simple, Flexible, Analytic, Arbitrage Free Volatility Interpolation. Being interested in the subject, I had a detailed look at it. It turns out that Thomas stumbled upon spline stochastic collocation without realizing it.

There are a few differences in his approach:

The optimization is on the x’s instead of the y’s, meaning the strike axis is fixed.
An original approach to avoid spurious modes, although I would have liked more details on it, with concrete examples of the penalty. Penalties are often challenging to get right.
A nice analysis of the asymptotic behavior in the wings, showing that an exponentially quadratic form for extrapolation corresponds to linear slopes in implied variance.

I had also explored a fixed strike axis initially (even back in 2014 - I uploaded some old notes on SSRN around this). I found it to be somewhat unstable back then, but it may well be that I did not put enough efforts into it, especially since optimizing the y’s instead of the x’s allowed for a straightforward use of B-splines, which is very attractive. It is very direct to impose monotonicity with B-splines.

NUFFT and COS

Mon, 14 Jul 2025 17:27:00 +0000

Leif Andersen and Mark Lake recently proposed the use of Non-Uniform Fast Fourier Transform for option pricing via the characteristic function. Fourier techniques are most commonly used for pricing vanilla options under the Heston model, in order to calibrate the model. They can be applied to other models, typically with known characteristic function, but also with numerically solved characteristic function as in the rough Heston model, and to different kind of payoffs, for example variance/volatility swaps, options. The subject has been vastly explored already so what’s new with this paper?

Forward Variance Models and Calibration

Sat, 21 Jun 2025 17:27:00 +0000

The modern rough volatility models adopt a forward variance curve terminology (see for example this paper on a rational approximation for the rough Heston, or this presentation on affine forward variance models or this paper on affine forward variance models). In this form, the rough Heston model reads:

According to the litterature, the initial forward variance curve is typically built from the implied volatilities through the variance swap replication: for each maturity, the price of a newly issued variance swap is computed to this maturity, using the implied volatilities for this maturity. Then we may compute the forward variance by interpolating linearly the variance swap prices and differentiating. This leads to the plot below.

A Not So Great New Simulation Scheme for the Heston Model

Wed, 18 Jun 2025 22:27:00 +0000

I recently saw a news about a great new simulation scheme for the Heston model by Abi Jaber. The paper suggests it is better than the popular alternatives such as the QE scheme of Leif Andersen. Reading it quickly, perhaps too quickly, I had the impression it would be more accurate especially when the number of time-steps is small.

The scheme is simple to implement so I decided to spend a few minutes to try it out. I had some test example for the DVSS2X scheme pricing a vanilla at-the-money option with Heston parameters v0=0.04, kappa=0.5, theta=0.04, rho=-0.9, sigma=1.0, and a time to maturity of 10 years. I don’t remember exactly where those parameters come from, possibly from Andersen paper. My example was using 8 time-steps per year, which is not that much. And here are the results with 1M paths (using scrambled Sobol):

Expansions for the time-dependent Heston model: Useless?

Sun, 01 Jun 2025 17:27:00 +0000

Fabrice Rouah wrote two books on the Heston model: one with C# and Matlab code, and one with VBA code. The two books are very similar. They are good in that they tackle most of the important points with the Heston model, from calibration to simulation. The calibration part (chapter 6) is a bit too short, it would have been great if it presented the actual difficulties with calibration in practice and went more in-depth with the techniques.

A New Asian Basket Spread Option Approximation

Tue, 22 Apr 2025 12:56:42 +0100

Around 10 years ago, while reading the excellent paper of Etore and Gobet on stochastic Taylor expansions for the pricing of vanilla options with discrete (cash) dividends, I had the idea of a small improvement, by using a more precise proxy for the Taylor expansion.

More recently, I applied the idea to approximate arithmetic Asian options prices by using the geometric Asian option price as a proxy (with some adjustments). This worked surprisingly well, and is competitive with the best implementations of Curran approach to Asian options pricing. I quickly noticed that I could apply the same idea to approximate Basket option prices, and from it obtain another approximation for vanilla options with cash dividends through the mapping described by J. Healy. Interestingly the resulting approximation is the most accurate amongst all other approximations for cash dividends.

LLMs Are Not Good For Basic Homework

Thu, 06 Mar 2025 12:56:42 +0100

My 12 years old child recently had a presentation to do with 3 other children for school, the subject was food and health. It was particularly difficult to organize the children and one of the children came with a full presentation with text, generated by an LLM (possibly corrected by the parents). At first, I thought this was just cheating and completly useless. Furthermore, he failed to read properly the details of the subject, and much of it wasn’t relevant.

Calibrating Heston to Variance Swaps - a bad idea?

Tue, 11 Feb 2025 12:56:42 +0100

An interesting idea to calibrate the Heston model in a more stable manner and reduce the calibration time is to make use of variance swap prices. Indeed, there is a simple formula for the theoretical price of a variance swap in the Heston model.

It is not perfect since it approximates the variance swap price by the expectation of the integrated variance process over time. In particular it does not take into account eventual jumps (obviously), finiteness of replication, and discreteness of observations. But it may be good enough. Thanks to this formula, we can calibrate three parameters of the Heston model: the initial variance, the long-term mean variance, and the speed of mean reversion to the term-structure of variance swaps. We do not need market prices of variance swaps, we may simply use a replication based on market vanilla options prices, such as the model-free replication of Fukasawa.

OpenSuse Tumbleweed to the Rescue

Thu, 06 Feb 2025 12:56:42 +0100

I recently upgraded a desktop computer, and to my surprise, the new motherboard was not fully supported by most Linux distributions. The main culprit was the network adapter, although the secure boot setup gave me lots of troubles as well. I had only a small usb key (2GB) and most (all?) live distributions do not fit on 2GB anymore. With the exception of Ubuntu images, I did not manage to boot from USB hard drive where I dumped the live image ISO, due to Secure boot related issues, even when disabling the feature or changing the settings in the BIOS. Using the small USB key worked with Secure boot enabled only.

Saving Us - Book Review

Sun, 13 Oct 2024 12:56:42 +0100

I had the opportunity to receive a free book on climate change, through the company I am working for. I had not heard of that book before, it called Saving Us and is written by an actual climate scientist (Katharine Hayhoe). Unfortunately, written by does not mean that it is a scientific book, and it’s not. The author does not spend much effort explaining the physics or the reports, but focuses on how to convince people this is an important problem to tackle. It is mildly interesting at first, as it presents the problem from a psychological angle. But it becomes quickly repetitive. The method is always the same, connect with what’s important to an audience who initially rejects climate warming (mostly the phenomenon, sometimes proposals around it), and make them understand that climate warming plays a role in their life, in the very matters they care about. It is presented as a series of personal experiences of the author, a list of examples.

Monotonicity of the Black-Scholes Option Prices in Practice

Sun, 29 Sep 2024 12:56:42 +0100

It is well known that vanilla option prices must increase when we increase the implied volatility. Recently, a post on the Wilmott forums wondered about the true accuracy of Peter Jaeckel implied volatility solver, whether it was truely IEEE 754 compliant. In fact, the author noticed some inaccuracy in the option price itself. Unfortunately I can not reply to the forum, its login process does not seem to be working anymore, and so I am left to blog about it.

Copilot vs ChatGPT on the Optimal Finite Difference Step-Size

Thu, 25 Jul 2024 12:56:42 +0100

When computing the derivative of a function by finite difference, which step size is optimal? The answer depends on the kind of difference (forward, backward or central), and the degree of the derivative (first or second typically for finance).

For the first derivative, the result is very quick to find (it’s on wikipedia). For the second derivative, it’s more challenging. The Lecture Notes of Karen Kopecky provide an answer. I wonder if Copilot or ChatGPT would find a good solution to the question:

News on the COS Method Truncation

Mon, 13 May 2024 20:56:42 +0100

The COS method is a fast way to price vanilla European options under stochastic volatility models with a known characteristic function. There are alternatives, explored in previous blog posts. A main advantage of the COS method is its simplicity. But this comes at the expense of finding the correct values for the truncation level and the (associated) number of terms.

A related issue of the COS method, or its more fancy wavelet cousin the SWIFT method, is to require a huge (>65K) number of points to reach a reasonable accuracy for some somewhat extreme choices of Heston parameters. I provide an example in a recent paper (see Section 5).

Variance Swap Term-Structure under Schobel-Zhu

Tue, 26 Mar 2024 12:56:42 +0100

I never paid too much attention to it, but the term-structure of variance swaps is not always realistic under the Schobel-Zhu stochastic volatility model.

This is not fundamentally the case with the Heston model, the Heston model is merely extremely limited to produce either a flat shape or a downward sloping exponential shape.

Under the Schobel-Zhu model, the price of a newly issued variance swap reads $$ V(T) = \left[\left(v_0-\theta\right)^2-\frac{\eta^2}{2\kappa}\right]\frac{1-e^{-2\kappa T}}{2\kappa T}+2\theta(v_0-\theta)\frac{1-e^{-\kappa T}}{\kappa T}+\theta^2+\frac{\eta^2}{2\kappa},,$$ where $ \eta $ is the vol of vol.

New Basket Expansions and Cash Dividends

Sat, 23 Mar 2024 09:56:42 +0100

In the previous post, I presented a new stochastic expansion for the prices of Asian options. The stochastic expansion is generalized to basket options in the paper, and can thus be applied on the problem of pricing vanilla options with cash dividends.

I have updated the paper with comparisons to more direct stochastic expansions for pricing vanilla options with cash dividends, such as the one of Etoré and Gobet, and my own refinement on it.

New Approximations for the Prices of Asian and basket Options

Sun, 17 Mar 2024 12:56:42 +0100

Many years ago, I had applied the stochastic expansion technique of Etore and Gobet to a refined proxy, in order to produce more accurate prices for vanilla options with cash dividends under the Black-Scholes model with deterministic jumps at the dividend dates. Any approximation for vanilla basket option prices can also be applied on this problem, and the sophisticated Curran geometric conditioning was found to be particularly competitive in The Pricing of Vanilla Options with Cash Dividends as a Classic Vanilla Basket Option Problem.

Easy Mistake With the Log-Euler Discretization On Black-Scholes

Mon, 11 Mar 2024 20:56:42 +0100

In the Black-Scholes model with a term-structure of volatilities, the Log-Euler Monte-Carlo scheme is not necessarily exact.

This happens if you have two assets $S_1$ and $S_2$, with two different time varying volatilities $\sigma_1(t), \sigma_2(t) $. The covariance from the Ito isometry from $t=t_0$ to $t=t_1$ reads $$ \int_{t_0}^{t_1} \sigma_1(s)\sigma_2(s) \rho ds, $$ while a naive log-Euler discretization may use $$ \rho \bar\sigma_1(t_0) \bar\sigma_2(t_0) (t_1-t_0). $$ In practice, the $ \bar\sigma_i(t_0) $ are calibrated such that the vanilla option prices are exact, meaning $$ \bar{\sigma}_i^2(t_0)(t_1-t_0) = \int_{t_0}^{t_1} \sigma_i^2(s) ds.$$

Roughness of Pure Jumps

Mon, 18 Dec 2023 20:56:42 +0100

In my previous blog post, I looked at the roughness of the SVCJ stochastic volatility model with jumps (in the volatility). In this model, the jumps occur randomly, but at discrete times. And with typical parameters used in the litterature, the jumps are not so frequent. It is thus more interesting to look at the roughness of pure jump processes, such as the CGMY process.

The CGMY process is more challenging to simulate. I used the approach based on the characteristic function described in Simulating Levy Processes from Their Characteristic Functions and Financial Applications. Ballota and Kyriakou add some variations based on FFT pricing of the characteristic function in Monte Carlo simulation of the CGMY process and option pricing and pay much care about a proper truncation range. Indeed, I found that the truncation range was key to simulate the process properly and not always trivial to set up especially for $Y \in (0,1) $. I however did not implement any automated range guess as I am merely interested in very specific use cases, and I used the COS method instead of FFT.

Roughness of Stochastic Volatility with Jumps

Thu, 07 Dec 2023 20:56:42 +0100

I was wondering if adding jumps to stochastic volatility, as is done in the SVCJ model of Duffie, Singleton and Pan “Transform Analysis and Asset Pricing for Affine Jump-Diffusion” also in Broadie and Kaya “Exact simulation of stochastic volatility and other affine jump diffusion processes”, would lead to rougher paths, or if it would mislead the roughness estimators.

The answer to the first question can almost be answered visually:

Measuring Roughness with Julia

Tue, 07 Nov 2023 20:56:42 +0100

I received a few e-mails asking me for the code I used to measure roughness in my preprint on the roughness of the implied volatility. Unfortunately, the code I wrote for this paper is not in a good state, it’s all in one long file line by line, not necessarily in order of execution, with comments that are only meaningful to myself.

In this post I will present the code relevant to measuring the oxford man institute roughness with Julia. I won’t go into generating Heston or rough volatility model implied volatilities, and focus only on the measure on roughness based on some CSV like input. I downloaded the oxfordmanrealizedvolatilityindices.csv from the Oxford Man Institute website (unfortunately now discontinued, data bought by Refinitiv but still available in some github repos) to my home directory

Black with Bachelier

Tue, 03 Oct 2023 20:56:42 +0100

I was experimenting with the recent SABR basket approximation of Hagan. The approximation only works for the normal SABR model, meaning beta=0 in SABR combined with the Bachelier option formula.

I was wondering how good the approximation would be for two flat smiles (in terms of Black volatilities). I then noticed something that escaped me before: the normal SABR model is able to fit the pure Black model (with constant vols) extremely well. A calibration near the money stays valid very out-of-the-money and the error in Black volatilities is very small.

Clenshaw-Curtis Quadrature Implementation by FFT in Practice

Wed, 27 Sep 2023 20:56:42 +0100

The Clenshaw-Curtis quadrature is known to be competitive with Gauss quadratures. It has several advantages:

the weights are easy and fast to compute.
adaptive / doubling quadratures are possible with when the Chebyshev polynomial of the second kind is used for the quadrature.
the Chebyshev nodes may also be used to interpolate some costly function.

The wikipedia article has a relatively detailed description on how to compute the quadrature weights corresponding to the Chebyshev polynomial of the second kind (where the points -1 and 1 are included), via a type-I DCT. It does not describe the weights corresponding to the Chebyshev polynomials of the first kind (where -1 and 1 are excluded, like the Gauss quadratures). The numbersandshapes blog post describes it very nicely. There are some publications around computation of Clenshaw-Curtis or Fejer rules, a recent one is Fast construction of Fejér and Clenshaw–Curtis rules for general weight functions.

Ghost Vacations

Sun, 20 Aug 2023 20:56:42 +0100

During my vacation, I don’t know why, but I looked at some stability issue with ghost points and the explicit method. I was initially trying out ghost points with the explicit runge kutta Chebyshev/Legendre/Gegenbauer technique and noticed some explosion in some cases.

I cornered it down to a stability issue of the standard explicit Euler method with ghost (or fictitious) points. The technique is described in the book “Paul Wilmott on Quantitative Finance” (also in Paul Wilmott introduces quantitative finance), which I find quite good, although I have some friends who are not much fond of it. The technique may be used to compute the price of a continuously monitored barrier option when the barrier does not fall on the grid, or more generally for time-dependent barriers. I however look at it in the simple context of a constant barrier in time.

Maximum Implied Variance Slope

Mon, 22 May 2023 23:56:42 +0100

The paper The Moment Formula for Implied Volatility at Extreme Strikes by Roger Lee redefined how practioners extrapolate the implied volatility, by showing that the total implied variance can be at most linear in the wings, with a slope below 2.

Shortly after, the SVI model of Jim Gatheral, with its linear wings, started to become popular.

In a recent paper in collaboration with Winfried Koller, we show that the asymptotic bounds are usually overly optimistic. This is somewhat expected, as the bound only holds as the log-moneyness goes to infinity. What is less expected, is that, even at very large strikes (nearly up to the limit of what SVI allow numerically), we may happen to have arbitrages, even though the bounds are respected.

The Return of the Arbitrage in the Perfect Volatility Surface

Wed, 29 Mar 2023 23:56:42 +0100

In a Wilmott article from 2018 (Wilmott magazine no. 97) titled “Arbitrage in the perfect volatility surface”, Uwe Wystup points out some interesting issues on seemingly innocuous FX volatility surfaces:

a cubic spline tends to produce artificial peaks/modes in the density.
SVI not arbitrage-free even on seemingly trivial input.

The examples provided are indeed great and the remarks very valid. There is more to it however:

a cubic spline on strikes or log-moneyness does not produce the artificial peak.
SVI with a=0 is arbitrage-free on this example.

For the first point, the denominator of the Dupire formula in terms of the representation of total variance as a function of logmoneyness gives some clues as it constitues a vega scaled version of the probability density with direct link to the total variance and its derivatives. In particular it is a simple function of its value, first and second derivatives, without involving any non-linear function and the second derivative only appears as a linear term. As such a low order polynomial representation of the variance in log-moneyness may be adequate. In contrast, the delta based representation introduces a strong non-linearity with the cumulative normal distribution function.

Splines on AUD/NZD 1w options.

Princeton Fintech and Quant conference of December 2022

Sun, 04 Dec 2022 07:56:42 +0100

I recently presented my latest published paper On the Bachelier implied volatility at extreme strikes at the Princeton Fintech and Quant conference. The presenters were of quite various backgrounds. The first presentations were much more business oriented with lots of AI keywords, but relatively little technical content while the last presentation was about parallel programming. Many were a pitch to recruit to employees.

The diversity was interesting: it was refreshing to hear about quantitative finance from vastly different perspectives. The presentation from Morgan Stanley about their scala annotation framework to ease up parallel programming was enlightening. The main issue they were trying to solve is the necessity for all the boilerplate code to handle concurrency, caching, robustness, which obfuscates significantly the business logic in the code. This is an old problem. Decades ago, rule engines were the trend for similar reasons. Using the example of pricing bonds, the presenters put very well forward the issues in evidence, issues that I found very relevant.

Desktop Linux in 2022

Wed, 26 Oct 2022 07:56:42 +0100

I have been a long time user of Fedora at work and have been happy quite happy about it. Around 6 months ago, I moved to Manjaro under VirtualBox in a Windows host, because the company imposes the use of a VPN client that does not run on Linux. It’s much less great for several reasons:

VirtualBox makes everything graphics much slower. It is still way better than WSL2 as I found WSL2 to be a poor man’s Linux, where graphics is via some half broken X11 client for Windows. With VirtualBox, programming is ok, watching videos is not (you need to use the host for that). Something relatively neat is that the VM can be run on any other host, including a Linux host by just copying the vm file.
I picked Manjaro out of curiosity and because it was perhaps faster/more optimized than Fedora. I don’t have too many issues with it. But I found out that the use of AUR packages would frequently break on upgrades. Futhermore, it is updated less frequently than Fedora, which is a bit surprising for a rolling release. I also installed it with ext4. With hindsight, I think both choices were poor, Fedora with brtfs is the way to go, or some Ubuntu variant.
Work also imposes various antivirus programs which tend to always keep the cpu busy.

Overall I am still much more productive than with a raw Windows, which is surprisingly slow for many software development related tasks. Having to deal more with Windows showed me how slow the system is, and how poor the experience is. Before using it again (after many years completely off it), I had started thinking it was good (well, it does crash much less than back in the days).

Roughness of the Implied Volatility

Sat, 09 Jul 2022 07:56:42 +0100

This is a follow up of my previous post on rough volatility. I recently tried to reproduce the results of the paper Rough Volatility: Fact or Artefact? as I was curious to apply the technique using different inputs. The 5-minutes SPX realized volatility is freely available in CSV format at the Oxford-Man Institute of Quantitative Finance and it is thus relatively straightforward to reproduce the numbers presented in the paper.

Using a sampling of K=75 and L=75*75, I obtain an rounghness index H=0.181. The paper uses K=70 and L = 70x70, and their Figure 19 of the paper states 0.187. It turns out that there are more than L observations in the time-series, and, with K=70, I end up with a roughness index H=0.222 when I start from the first observation (year 2000), up to the observation L+1. But it is possible to slide this window and compute the roughness index at each starting point. The results are enlightening.

SPX historical vol roughness index sliding estimate with K=70.

density of SPX historical vol roughness index estimate with K=70. The mean is 0.158 with a standard deviation of 0.034

Volatility: Rough or Not? A Short Review

Tue, 10 May 2022 17:56:42 +0100

It is well-known that the assumption of constant volatility in the Black-Scholes model for pricing financial contracts is wrong and may lead to serious mispricing, especially for any exotic derivative contracts. A classic approach is to use a deterministic local volatility model to take into account the variation both in the time dimension and in the underlying asset price dimension. But the model is still not realistic in terms of forward smile (the implied volatilities of forward starting options). A stochastic volatility component must be added to correct for it. More recently, the concept of rough volatility emerged in many academic papers. Instead of using a classic Brownian motion for the stochastic volatility process, a fractional Brownian motion is used. The idea of using a fractional Brownian motion for financial time-series can be traced back to Mandelbrot, but it is only relatively recently that it has been proposed for the volatility process (and not the stock process).

Monte-Carlo Parallelization: to vectorize or not?

Sat, 09 Apr 2022 21:56:42 +0100

When writing a Monte-Carlo simulation to price financial derivative contracts, the most straightforward is to code a loop over the number of paths, in which each path is fully calculated. Inside the loop, a payoff function takes this path to compute the present value of the contract on the given path. The present values are recorded to lead to the Monte-Carlo statistics (mean, standard deviation). I ignore here any eventual callability of the payoff which may still be addressed with some work-arounds in this setup. The idea can be schematized by the following go code:

for i := 0; i < numSimulations; i++ {
	pathGenerator.ComputeNextPath(&path) //path contains an array of n time-steps of float64.
	pathEvaluator.Evaluate(&path, output)
	statistics.RecordValue(output)
}

A python programmer would likely not write a simulation this way, as the python code inside the large loop will not be fast. Instead, the python programmer will write a vectorized simulation, generating all paths at the same time.

pathGenerator.ComputeAllPaths(&paths) //paths contains an array of n time-steps of vectors of size numSimulations 
pathEvaluator.EvaluateAll(&paths, output)
statistics.RecordValues(output)

More Automatic Differentiation Awkwardness

Tue, 04 Jan 2022 21:56:42 +0100

This blog post from Jherek Healy presents some not so obvious behavior of automatic differentiation, when a function is decomposed into the product of two parts where one part goes to infinity and the other to zero, and we know the overall result must go to zero (or to some other specific number). This decomposition may be relatively simple to handle for the value of the function, but becomes far less trivial to think of in advance, at the derivative level.

Quadprog in Julia

Sun, 21 Nov 2021 13:56:42 +0100

As described on wikipedia, a quadratic programming problem with n variables and m constraints is of the form $$ \min(-d^T x + 1/2 x^T D x) $$ with the constraints $ A^T x \geq b_0 $, were $D$ is a $n \times n$-dimensional real symmetric matrix, $A$ is a $n \times m$-dimensional real matrix, $ b_0 $ is a $m$-dimensional vector of constraints, $ d $ is a $n$-dimensional vector, and the variable $x$ is a $n$-dimensional vector.

Positive Stochastic Collocation

Tue, 31 Aug 2021 07:56:42 +0100

In the context of my thesis, I explored the use of stochastic collocation to capture the marginal densities of a positive asset. Indeed, most financial asset prices must be non-negative. But the classic stochastic collocation towards the normally distributed random variable, is not.

A simple tweak, proposed early on by Grzelak, is to assume absorption and use the put-call parity to price put options (which otherwise depend on the left tail). This sort of works most of the time, but a priori, there is no guarantee that we will end up with a positive put option price. As an extreme example, we may consider the case where the collocation price formula leads to $V_{\textsf{call}}(K=0) < f $ where $f $ is the forward price to maturity. The put-call parity relation applied at $K=0 $ leads to $V_{\textsf{put}}(K=0) = V_{\textsf{call}}(K=0)-f < 0 $. This means that for some strictly positive strike, the put option price will be negative, which is non-sensical. In reality, it thus implies that absorption must happen earlier, not at $S=0 $, but at some strictly positive asset price. And then it is not so obvious to chose the right value in advance.

My Review of Steven Koonin's Book on Climate

Mon, 30 Aug 2021 07:56:42 +0100

Steven Koonin, who was Secretary for Science, Department of Energy, in the Obama administration recently wrote a somewhat controversial book on climate science with the title Unsettled. I was curious to read what kind of critics a physicist who partly worked in the field had, even if I believe that climate warming is real, and humans have an influence on it. It turns out that some of his remarks regarding models are relevant way beyond climate science, but some other subjects are not as convincing.

Github and SSH setup

Mon, 16 Aug 2021 07:56:42 +0100

Github recently moved to support only ssh access via public/private keys. As I use github to host this blog, I was impacted.

The setup on Linux is not very complicated, and relatively well documented on Github itself but all the steps are not listed in a simplistic manner, and some Google search is still required to find out how to setup multiple private keys for various different servers or different repos.

Covid-19 Fake News

Tue, 10 Aug 2021 07:56:42 +0100

One thing that motivated me for vaccination is the fake news propaganda against the Covid-19 vaccines.

A mild example relates to the data from Israel about the delta variant. This kind of article, with the title “Covid 19 Case Data in Israel, a Troubling Trend”, puts emphasis on the doubts on the effectivness of the vaccine:

the vaccine appears to have a negligible effect on an individual as to whether he/she catches the current strain. Moreover, the data indicates that the current vaccines used (Moderna, Pfizer-BioNTech, AstraZeneca) may have a decreasing effect on reduced hospitalizations and death if one does get infected with the Delta variant.

Are traditional banks ready for the 21st century?

Sat, 12 Jun 2021 07:56:42 +0100

This is a follow up on my parents phishing scam.

After several weeks, my parents and I were finally able to have a real world meeting with the advisor at the bank. The advisor is a young woman with an obvious background in sales.

In order to process the paperwork around the reimbursement of the phishing scam, the main issue was the request of the original phishing e-mail by the bank, as my mother had deleted the e-mail. It turns out, that in Thunderbird, deleted e-mails are not deleted on disk until an operation of compactification of the mailbox is done. I was thus able to recover the deleted e-mail. Interestingly, the deleted e-mail was not in the trash file, but directly in the inbox file.

Yesterday, Pirates Took Over My Parents Bank Account

Wed, 10 Mar 2021 07:56:42 +0100

This is the story of their hack.

Yesterday evening, I received a call from my mother, frantic over the phone. She says she sees alerts of withdrawals from her bank account on her phone, with new alerts every 5 minutes or so. I try to ask her if she clicked recently on some e-mail related to her bank. She is so panicked that I don’t manage to have an answer. While trying to understand if those alerts are real or not, my wife suggests immediately that my mother should call her bank. On the phone, I ask

Remarkable Coincidences, Bad Book?

Sat, 21 Nov 2020 07:56:42 +0100

I stumbled upon a new short book Financial Models in Production from O. Kettani and A. Reghai. A page attracted my attention

A page from Kettani and Reghai's book.

This is the same example as I used on my blog, where I also present the Li’s SOR method combined with the good initial guess from Stefanica. The idea has also been expanded on in Jherek Healy’s book. What is shocking is that, beside reusing my example, they reuse my timing for Jäckel and my implementation is in Google Go, with a timing done on some older laptop. The numbers given are thus highly inconsistent. Of course, none of this is mentioned anywhere, and the book does not reference my blog.

Bad papers and the roots of high degree polynomials

Sat, 07 Nov 2020 07:56:42 +0100

I was wondering what were exactly the eigenvalues of the Mersenne-Twister random number generator transition matrix. An article by K. Savvidy sparked my interest on this. This article mentioned a poor entropy (sum of log of eigenvalues amplitudes which are greater than 1), with eigenvalues falling almost on the unit circle.

The eigenvalues are also the roots of the characteristic polynomial. It turns out, that for jumping ahead in the random number sequence, we use the characteristic polynomial. There is a twist however, we use it in F2 (modulo 2), for entropy, we are interested in the characteristic polynomial in Z (no modulo), specified in Appendix A of the Mersenne-Twister paper. The roots of the two polynomials are of course very different.

Disaster Capitalism - Summer Reading Review

Fri, 06 Nov 2020 20:56:42 +0100

Several years ago, I read the book No Logo from Naomi Klein. I did not find it particularly good, but it did raise a valid concern overall. This summer I read Shock Therapy - The rise of disaster capitalism. It suffers from some of the same flaws as No Logo, namely a lot of repetition of the same idea. Here, the underlying idea is that neoliberalism does not work in practice, and often ends up being some kind of corporatism. At the same time, it is suggested that some mild socialism is often much better for the people, although, the latter is not backed by concrete examples in the book. The former is backed by numerous documents, and is analyzed accross time and countries. It starts with Chili under Pinochet, the prototypical example that force is required to impose neoliberalism, then moves around South America in general, with some cases where a strong inflation, may be enough for the people to accept neoliberalism. Then it continues with China under Deng Xiao Ping, which I find a bit too much of a stretch to make a case about any kind of neoliberalism. Russia under Yeltsin is next, and it ends with the war in Irak and the USA.

More on random number generators

Sat, 10 Oct 2020 20:56:42 +0100

My previous post described the recent view on random number generators, with a focus on the Mersenne-Twister war.

Since, I have noticed another front in the war of the random number generators:

An example in dimension 121 from K Savvidy where L’Ecuyer MRG32k3a fails to compute the correct result, regardless of the seed. This is a manufactured example, such that the vector, used in the example, falls in the dual lattice of the generator. Similar examples can be constructed for other variants.
Spectral Analysis of the MIXMAX Random Number Generator (by L’Ecuyer et al.) shows defects in MIXMAX, for some parameters that were advised in earlier papers of K. Savvidy. MIXMAX is a RNG popularized by K. Savvidy, used in CLHEP at Fermilab.

Also, I found interesting that Monte-Carlo simulations run at the Los Alamos National Laboratory relied on a relatively simple linear congruential generator (LCG) producing 24- or 48-bits integers for at least 40 years. LCGs are today touted as some of the worst random number generators, exhibiting strong patterns in 2D projections. Also the period chosen was very small by today’s standards: 7E13.

The war of the random number generators

Thu, 17 Sep 2020 20:56:42 +0100

These days, there seems to be some sort of small war to define what is a modern good random number generators to advise for simulations. Historically, the Mersenne-Twister (MT thereafter) won this war. It is used by default in many scientific libraries and software, even if there has been a few issues with it:

A bad initial seed may make it generate a sequence of low quality for at least as many as 700K numbers.
It is slow to jump-ahead, making parallelization not so practical.
It fails some TestU01 Bigcrush tests, mostly related to the F2 linear algebra, the algebra of the Mersenne-Twister.

It turns out, that before MT (1997), a lot of the alternatives were much worse, except, perhaps, RANLUX (1993), which is quite slow due to the need of skipping many points of the generated sequence.

Sobol with 64-bits integers

Wed, 09 Sep 2020 20:56:42 +0100

A while ago, I wondered how to make some implementation of Sobol support 64-bits integers (long) and double floating points. Sobol is the most used quasi random number generator (QRNG) for (quasi) Monte-Carlo simulations.

The standard Sobol algorithms are all coded with 32-bits integers and lead to double floating point numbers which can not be smaller than $ 2^{-31} $. I was recently looking back at the internals at Sobol generators, and noticed that generating with 64-bits integers would not help much.

Intel failure and the future of computing

Fri, 24 Jul 2020 20:56:42 +0100

What has been happening to the INTC stock today may be revealing of the future. The stock dropped more than 16%, mainly because they announced that their 7nm process does not work (well) and they may rely on an external foundry for their processors. Initially, in 2015, they thought they would have 8nm process by 2017, and 7nm by 2018. They are more than 3 years late.

Intel used to be a leader in the manufacturing process for microprocessor. While the company has its share of internal problems, it may also be that we are starting to hit the barrier, where it becomes very difficult to improve on the existing. The end of Moore’s law has been announced many times, it was already a subject 20 years ago. Today, it may be real, if there is only a single company capable of manufacturing processor using a 5nm process (TSMC).

March 9, 2020 crash - where will CAC40 go?

Mon, 09 Mar 2020 21:56:42 +0100

The stock market crashed by more than 7% on March 9, 2020. It is one of the most important drop since September 2001. I looked at BNP warrant prices on the CAC40 French index, with a maturity of March 20, 2020 , to see what they would tell about the market direction on the day of the crash. This is really a not-so-scientific experiment.

The quotes I got were quite noisy. I applied a few different techniques to imply the probability density from the option prices:

42

Thu, 05 Dec 2019 20:56:42 +0100

Today my 6-years old son came with a math homework. The stated goal was to learn the different ways to make 10 out of smaller numbers. I was impressed. Immediately, I wondered

how many ways are there to make 10 out of smaller numbers?

This is one of the beauties of maths: a very simple problem, which a 6-years old can understand, may actually be quite fundamental. If you want to solve this in the general case, for any number instead of 10, you end up with the partition function. And in order to find this, you will probably learn recurrence relations. So what is the answer for 10?

Numba, Pypy Overrated?

Tue, 12 Feb 2019 20:56:42 +0100

Many benchmarks show impressive performance gains with the use of Numba or Pypy. Numba allows to compile just-in-time some specific methods, while Pypy takes the approach of compiling/optimizing the full python program: you use it just like the standard python runtime. From those benchmarks, I imagined that those tools would improve my 2D Heston PDE solver performance easily. The initialization part of my program contains embedded for loops over several 10Ks elements. To my surprise, numba did not improve anything (and I had to isolate the code, as it would not work on 2D numpy arrays manipulations that are vectorized). I surmise it does not play well with scipy sparse matrices. Pypy did not behave better, the solver became actually slower than with the standard python interpreter, up to twice as slow, for example, in the case of the main solver loop which only does matrix multiplications and LU solves sparse systems. I did not necessarily expect any performance improvement in this specific loop, since it only consists in a few calls to expensive scipy calculations. But I did not expect a 2x performance drop either.

About

Mon, 17 Dec 2018 10:05:07 +0100

I just moved my blog to a static website, created with Hugo, I explain the reasons why here. You can find more about me on my linkedin profile. In addition you might find the following interesting:

List of quantitative finance papers I have freely available on SSRN
TR-BDF2 for Stable American Option Pricing. This is the first draft, not the final version that was published in the Journal of Computational Finance.
Presentation on finite difference techniques for arbitrage-free SABR I gave at the conference on models and numerics in financial mathematics at the Lorentz center in 2015.
Exact Forward for Finite-Difference Schemes on the Log-transformed Black-Scholes PDE.
Asymptotic bounds of the normal volatility.

Fixing NaNs in Quadprog

Sun, 07 Oct 2018 20:56:42 +0100

Out of curiosity, I tried quadprog as open-source quadratic programming convex optimizer, as it is looks fast, and the code stays relatively simple. I however stumbled on cases where the algorithm would return NaNs even though my inputs seemed straighforward. Other libraries such as CVXOPT did not have any issues with those inputs.

Searching on the web, I found that I was not the only one to stumble on this kind of issue with quadprog. In particular, in 2014, Benjamen Tyner gave a simple example in R, where solve.QP returns NaNs while the input is very simple: an identity matrix with small perturbations out of the diagonal. Here is a copy of his example:

On the Probability of a Netflix Stock Crash

Thu, 12 Jul 2018 20:56:42 +0100

This is a follow up to my previous post where I explore the probability of a TSLA stock crash, reproducing the results of Timothy Klassen.

According to the implied cumulative probability density, TSLA has around 15% chance of crashing below $100. Is this really high compared to other stocks? or is it the interpretation of the data erroneous?

Here I take a look at NFLX (Netflix). Below is the implied volatility according to three different models.

On the Probability of a TSLA Stock Crash

Wed, 11 Jul 2018 20:56:42 +0100

Timothy Klassen had an interesting post on linkedin recently, with the title “the options market thinks there is a 16% chance that Tesla will not exist in January 2020”. As I was also recently looking at the TSLA options, I was a bit intrigued. I looked at the option chain on July 10th, and implied the European volatility from the American option prices. I then fit a few of my favorite models: Andreasen-Huge with Tikhonov regularization, the lognormal mixture, and a polynomial collocation of degree 7. This results in the following graph

The Fourth Moment of the Normal SABR Model

Mon, 11 Jun 2018 20:56:42 +0100

I was wondering if I could use the SABR moments to calibrate a model to SABR parameters directly. It turns out that the SABR moments have relatively simple expressions when $\beta=0$, that is, for the normal SABR model (with no absorption). This is for the pure SABR stochatic volatility model, not the Hagan approximation. For the Hagan approximation, we would need to use the replication by vanilla options to compute the moments.

Implying the Probability Density from Market Option Prices (Part 2)

Sun, 27 May 2018 20:56:42 +0100

This is a follow-up to my posts on the implied risk-neutral density (RND) of the SPW options before and after the big volatility change that happened in early February with two different techniques: a smoothing spline on the implied volatilities and a Gaussian kernel approach.

The Gaussian kernel (as well as to some extent the smoothing spline) let us believe that there are multiple modes in the distribution (multiple peaks in the density). In reality, Gaussian kernel approaches will, by construction, tend to exhibit such modes. It is not so obvious to know if those are real or artificial. There are other ways to apply the Gaussian kernel, for example by optimizing the nodes locations and the standard deviation of each Gaussian. The resulting density with those is very similar looking.

Senior Developers Don't Know OO Anymore

Thu, 08 Mar 2018 20:56:42 +0100

It has been a while since the good old object-oriented (OO) programming is not trendy anymore. Functional programming or more dynamic programming (Python-based) have been the trend, with an excursion in template based programming for C++ guys. Those are not strict categories: Python can be used in a very OO way, but it’s not how it is marketed or considered by the community.

Recently, I have seen some of the ugliest refactoring in my life as a programmer, done by someone with at least 10 years of experience programming in Java. It is a good illustration because the piece of code is particularly simple (although I won’t bother with implementation details). The original code was a simple boolean method on an object such as

Implying the Probability Density from Market Option Prices

Tue, 13 Feb 2018 20:56:42 +0100

In the previous post, I showed a plot of the probability implied from SPW options before and after the big volatility change of last week. I created it from a least squares spline fit of the market mid implied volatilities (weighted by the inverse of the bid-ask spread). While it looks reasonable, the underlying technique is not very robust. It is particularly sensitive to the number of options strikes used as spline nodes.

Where is the S&P 500 going to end?

Tue, 06 Feb 2018 19:56:42 +0100

Yesterday the American stocks went a bit crazy along with the VIX that jumped from 17.50 to 38. It’s not exactly clear why, the news mention that the Fed might raise its interest rates, the bonds yield have been recently increasing substantially, and the market self-correcting after stocks grew steadily for months in a low VIX environment.

I don’t exactly follow the SPX/SPW options daily. But I had taken a snapshot two weeks ago when the market was quiet. We can imply the probability density from the market option prices. It’s not an exact science. Here I do this with a least-squares spline on the implied volatilities (the least squares smoothes out the noise). I will show another approach in a subsequent post.

Discrete Sine Transform via the FFT

Mon, 05 Feb 2018 13:56:42 +0100

Several months ago, I had a quick look at a recent paper describing how to use Wavelets to price options under stochastic volatility models with a known characteristic function. The more classic method is to use some numerical quadrature directly on the Fourier integral as described in this paper for example. When I read the paper, I was skeptical about the Wavelet approach, since it looked complicated, and with many additional parameters.

Quantitative Finance Books Citing My Papers

Sat, 09 Dec 2017 13:56:42 +0100

I would have never really expected that when I started writing papers, but little by little there is a growing list of books citing my papers:

Applied Quantitative Finance for Equity Derivatives by Jherek Healy: the most recent book on equity derivatives refers to several of my papers. In contrast with many other books, the author goes beyond and provides additional insights on the papers.
Interest Rate Derivatives Explained: Volume 2: Term Structure and Volatility Modelling by Jörg Kienitz and Peter Casper. It refers to the paper “finite difference techniques for arbitrage-free SABR”, written in collaboration with Gary Kennedy.
Interest Rate Derivatives Explained: Volume 1: Products and Markets by Jörg Kienitz. It refers to my paper on curve interpolation (there is a mistake in the actual reference given inside the book,about arbitrage-free SABR, which unrelated to the text). I like how this book gives real world market data related to the products considered.
Interest Rate Modelling in the Multi-Curve Framework: Foundations, Evolution and Implementation by Marc Henrard. It refers to the paper about yield curve interpolation. This is one of the rare books to present curve construction in depth.

There are also some Springer books which are typically a collection of papers on a specific subject (which I find less interesting).

Google phones are overrated

Mon, 02 Oct 2017 23:56:42 +0100

It is a relatively common belief that the vanilla Android experience is better, as it runs smoother. The Samsung Touchwiz is often blamed for making things slow and not more practical.

I have had a Nexus 6 for a couple of years and I noticed the slowdowns after each update, up to a point where it sometimes took a few seconds to open the phone app, or to display the keyboard. I freed up storage, removed some apps but this did not make any difference. This is more or less the same experience that people have with the Samsung devices if reddit comments are to be believed.

SVN is dead

Tue, 26 Sep 2017 23:56:42 +0100

A few years ago, when Git was rising fast and SVN was already not hype anymore, a friend thought that SVN was for many organizations better suited than Git, with the following classical arguments, which were sound at the time:

Who needs decentralization for a small team or a small company working together?
‎SVN is proven, works well and is simple to use and put in place.

Each argument is in reality not so strong. It becomes clear now that Git is much more established.

The Neural Network in Your CPU

Sun, 06 Aug 2017 23:56:42 +0100

Machine learning and artificial intelligence are the current hype (again). In their new Ryzen processors, AMD advertises the Neural Net Prediction. It turns out this is was already used in their older (2012) Piledriver architecture used for example in the AMD A10-4600M. It is also present in recent Samsung processors such as the one powering the Galaxy S7. What is it really?

The basic idea can be traced to a paper from Daniel Jimenez and Calvin Lin “Dynamic Branch Prediction with Perceptrons”, more precisely described in the subsequent paper “Neural methods for dynamic branch prediction”. Branches typically occur in if-then-else statements. Branch prediction consists in guessing which code branch, the then or the else, the code will execute, thus allowing to precompute the branch in parallel for faster evaluation.

Benham disc in web canvas

Mon, 10 Jul 2017 23:56:42 +0100

Around 15 years ago, I wrote a small Java applet to try and show the Benham disk effect. Even back then applets were already passé and Flash would have been more appropriate. These days, no browser support Java applets anymore, and very few web users have Java installed. Flash also mostly disappeared. The web canvas is today’s standard allowing to embbed animations in a web page. This effect shows color perception from a succession of black and white pictures. It is a computer reproduction from the Benham disc with ideas borrowed from "Pour La Science Avril/Juin 2003". Using a delay between 40 and 60ms, the inner circle should appear red, the one in the middle blue and the outer one green. When you reverse the rotation direction, blue and red circles should be inverted.

Blogs on Quantitative Finance

Wed, 21 Jun 2017 23:56:42 +0100

There are not many blogs on quantitative finance that I read. Blogs are not so popular anymore with the advent of the various social networks (facebook, stackoverflow, google plus, reddit, …). Here is a small list:

Clarus FT: often interesting statistics on the swap market, clearing, plus the more technical articles from Gary.
Quants R Us: A relatively new blog with a promising starting post analyzing Andreasen-Huge one-step local-volatility algorithm with a Spline
Fooling around with Quantlib: the blog from Peter Caspers, also relevant to non-Quantlib professionals has original insights such as Smile dynamics by densities or the Supernatural Libor Coupons. Unfortunately it is not so active anymore.
Implementing Quantlib: the blog from Luigi Ballabio, which explains many of the design decisions in Quantlib. Very interesting for developer of financial libraries, see for example the fd solvers.
HPC Quantlib from Klaus Spanderen. Yes lots of quantlib blogs, but this one is actually not much focused on quantlib. It goes into great details about some numerical techniques, see for example the analysis of Heston pricing algorithm
Oxford Strat. It is a different kind of subject: too many trading strategies but some interesting data, for example global market correlations and ideas.
Wilmott forums not a blog, but it sometimes (not often) has interesting discussions and can be a good way to connect.

Another way to find out what’s going on in the quantitative finance world is to scan regularly recent papers on arxiv, SSRN or the suggestions of Google scholar.

Typo in Hyman non-negative constraint - 28 years later

Tue, 23 May 2017 23:56:42 +0100

In their paper “Nonnegativity-, Monotonicity-, or Convexity-Preserving Cubic and Quintic Hermite Interpolation”, Dougherty, Edelman and Hyman present a simple filter on the first derivatives to maintain positivity of a cubic spline interpolant.

Unfortunately, in their main formula for non-negativity, they made a typo: the equation (3.3) is not consistent with the equation (3.1): the $ \Delta x_{i-1/2} $ is interverted with $ \Delta x_{i+1/2} $.

It was not obvious to find out which equation was wrong since there is no proof in the paper. Fortunately, the proof is in the reference paper “Monotone piecewise bicubic interpolation” from Carlson and Fritsch and it is clear then that equation (3.1) is the correct one.

Implied Volatility from Black-Scholes price

Sun, 02 Apr 2017 07:56:42 +0100

Dan Stefanica and Rados Radoicic propose a quite good initial guess in their very recent paper An Explicit Implied Volatility Formula. Their formula is simple, fast to compute and results in an implied volatility guess with a relative error of less than 10%.

It is more robust than the rational fraction from Minquiang Li: his rational fraction is only valid for a fixed range of strikes and maturities. The new approximation is mathematically proved accurate across all strikes and all maturities. There is only the need to be careful in the numerical implementation with the case where the price is very small (a Taylor expansion of the variable C will be useful in this case).

The VIX starts smiling

Tue, 21 Mar 2017 07:56:42 +0100

The VIX implied volatilities used to look like a logarithmic function of the strikes. I don’t look at them often, but today, I noticed that the VIX had the start of a smile shape.

1m VIX implied volatilities on March 21, 2017 with strictly positive volume.

Very few strikes trades below the VIX future level (12.9). All of this is likely because the VIX is unusually low: not many people are looking to trade it much lower.

When SVI Breaks Down

Thu, 16 Mar 2017 07:56:42 +0100

In order to fit the implied volatility smile of equity options, one of the most popular parameterization is Jim Gatheral’s SVI, which I have written about before here.

It turns out that in the current market conditions, SVI does not work well for short maturities. SPX options expiring on March 24, 2017 (one week) offer a good example. I paid attention to include only options with non zero volume, that is options that are actually traded.

Brownian Bridge and Discrete Random Variables

Thu, 26 Jan 2017 14:55:32 +0100

The new Heston discretisation scheme I wrote about a few weeks ago makes use a discrete random variable matching the first five moments of the normal distribution instead of the usual normally distributed random variable, computed via the inverse cumulative distribution function. Their discrete random variable is: $$\xi = \sqrt{1-\frac{\sqrt{6}}{3}} \quad \text{ if } U_1 < 3,,$$ $$ \xi =-\sqrt{1-\frac{\sqrt{6}}{3}} \quad \text{ if } U_1 > 4,,$$ $$\xi = \sqrt{1+\sqrt{6}} \quad \text{ if } U_1 = 3,,$$ $$\xi = -\sqrt{1+\sqrt{6}} \quad \text{ if } U_1 = 4,,$$ with $U_1 \in \{0,1,…,7\}$

Samsung Wireless Printer under Fedora 25

Thu, 26 Jan 2017 09:55:32 +0100

This is a note for those who want to setup a Samsung wireless printer under Linux. It is quite simple, this forum post helped me, the actual useful steps on Fedora 25 are:

download tar.gz linux driver from Samsung website. As root, unpack & install:

tar xvzf SamsungPrinterInstaller.tar.gz 
cd uld
./install.sh

in the printer menu, lookup for the wireless key (8 digits),
connect to the printer Wifi network with a computer using the wireless key,
enter the network gateway IP, this is typically http://192.168.3.1,
click on Login with user “admin” and password “sec00000”,
open the network wifi settings and select “Easy Wi-Fi Settings”,
connect back to your wireless network,
add the network printer via the printer settings wizard, or alternatively through CUPS interface http://localhost:631.

That’s it. Printer and scanner will then work on your local Wi-Fi network.

A new scheme for Heston - Part 2

Mon, 23 Jan 2017 07:55:32 +0100

A couple weeks ago, I wrote about a new Heston discretisation scheme which was at least as accurate as Andersen QE scheme and faster, called DVSS2.

It turns out that it does not behave very well on the following Vanilla forward start option example (which is quite benign). The Heston parameters comes from a calibration to the market and are

$$v_0= 0.0718, \kappa= 1.542, \theta= 0.0762, \sigma= 0.582, \rho= -0.352$$

with a maturity of one year.

Power management and XFCE under Fedora 25

Thu, 19 Jan 2017 09:55:32 +0100

The gnome shell has been crashing on me more regularly lately. XFCE is a good and fast more tradional desktop, but, from past experiences, it does not play well with power management if you only install it via

dnf install @xfce-desktop-environment

My typical experience is a black screen after resuming from suspend (sometimes, not always), or hibernate (always) and most of the time I end up just rebooting. It turns out this is all caused by the interaction between the gdm login daemon and xfce. Moving to the lightdm login daemon instead fixes those issues for Fedora 25:

Equivalence between floating-strike and fixed-strike Asian options

Wed, 18 Jan 2017 09:55:32 +0100

Many papers present formulae to price Asian options in the Black-Scholes world only for the fixed strike Asian case, that is a contract that pays $ \max(A-K,0)$ at maturity $T$ where $A = \sum_{i=0}^{n-1} w_i S(t_i) $ is the Asian average.

More generally, this can be seen as the payoff of a Basket option where the underlyings are just the same asset but at different times. And any Basket option formula can actually be used to price fixed-strike Asian options by letting the correlation correspond to the correlation between the asset at the averaging times and the variances correspond to the variance at each averaging time. The basket approach allows then naturally for a term-structure of rates, dividends and volatilities.

Bachelier Normal Volatility Asymptotics

Tue, 17 Jan 2017 09:55:32 +0100

It is relatively well known that the Black-Scholes volatility can not grow faster than $\sqrt{\ln(K)}$. The rule is also sometimes simply stated as “the implied variance can not grow faster than linear” (in log-moneyness). The proof comes from Roger Lee “The moment formula for implied volatility at extreme strikes” but the rule was suggested earlier, for example in Hodge’s paper from 1996 “Arbitrage bounds of the implied volatility strike and term structures of European-style options”.

A new scheme for Heston

Fri, 06 Jan 2017 07:55:32 +0100

I stumbled recently upon a new Heston discretisation scheme, in the spirit of Alfonsi, not more complex and more accurate.

My first attempt at coding the scheme resulted in a miserable failure even though the described algorithm looked not too difficult. I started wondering if the paper, from a little known Lithuanian mathematical journal, was any good. Still, the math in it is very well written, with a great emphasis on the settings for each proposition.

Andreasen-Huge interpolation - Don't stay flat

Tue, 13 Dec 2016 09:55:32 +0100

Jesper Andreasen and Brian Huge propose an arbitrage-free interpolation method based on a single-step forward Dupire PDE solution in their paper Volatility interpolation. To do so, they consider a piecewise constant representation of the local volatility in maturity time and strike where the number of constants matches the number of market option prices.

An interesting example that shows some limits to the technique as described in Jesper Andreasen and Brian Huge paper comes from Nabil Kahale paper on an arbitrage-free interpolation of volatilities.

option volatilities for the SPX500 in October 1995.

Put-Call parity and the log-transformed Black-Scholes PDE

Mon, 05 Dec 2016 09:55:32 +0100

We will assume zero interest rates and no dividends on the asset $S$ for clarity. The results can be easily generalized to the case with non-zero interest rates and dividends. Under those assumptions, the Black-Scholes PDE is: $$ \frac{\partial V}{\partial t} + \frac{1}{2}\sigma^2 S^2 \frac{\partial^2 V}{\partial S^2} = 0.$$

An implicit Euler discretisation on a uniform grid in $S$ of width $h$ with linear boundary conditions (zero Gamma) leads to:

Benaim et al. extrapolation does not work on equities

Tue, 04 Oct 2016 09:55:32 +0100

We have seen earlier that a simple parabola allows to capture the smile of AAPL 1m options surprisingly well. For very high and very low strikes, the parabola does not obey Lee’s moments formula (the behavior in the wings needs to be at most linear in variance/log-moneyness).

Extrapolating the volatility smile in the low or high strikes in a smooth $C^2$ fashion is however not easy. A surprisingly popular so called “arbitrage-free” method is the extrapolation of Benaim, Dodgson and Kainth developed to remedy the negative density of SABR in interest rates as well as to give more control over the wings.

AES for Monte-Carlo

Wed, 17 Aug 2016 09:55:32 +0100

In finance, and also in science, the Mersenne-Twister is the de-factor pseudo-random number generator (PRNG) for Monte-Carlo simulations. By the way, there is a recent 64-bit maximally equidistributed version called MEMT19937 with 53-bit double precision floating point numbers in mind.

D.E. Shaw paper Parallel Random Numbers: As easy as 1, 2, 3 makes a bold remark: since specific AES instructions have been available since 2010 in most x86 processors, why not use them?

Historicaly, counter-based PRNGs based on cryptographic standards such as AES were historically slow, which motivated the development of sequential PRNGs with good statistical properties, yet not cryptographically strong like the Mersenne Twister for Monte-Carlo simulations.

A review of Thomas Piketty - Le capital au XXI siecle

Tue, 02 Aug 2016 09:55:32 +0100

I found back some old notes I had written about the book “Le capital au XXI siecle” from Thomas Piketty. It took me a while to finish that book last summer.

So many journalists have written around Piketty, that I had to buy the book and read it. It turns out that some of the criticism I have read is not really founded once one reads the book, but here are other real obvious criticisms that I suprisingly did not hear.

Number of regressors in a BSDE

Tue, 26 Jul 2016 09:55:32 +0100

Last year, I was kindly invited at the workshop on Models and Numerics in Financial Mathematics at the Lorentz center. It was surprinsgly interesting on many different levels. Beside the relatively large gap between academia and the industry, which this workshop was trying to address, one thing that struck me is how difficult it was for people of slightly different specialties to communicate.

It seemed that mathematicians of different countries working on different subjects related to backward stochastic differential equations (BSDEs) would not truly understand each other. I know this is very subjective, and maybe those mathematicians did not feel this at all. One concrete example is related to the number of regressors needed to solve a BSDE on 10 different variables. Solving a BSDE on 10 variables for EDF was given as an illustration at the end of an otherwise excellent presentation. Someone in the audience asked how possibly they could do that in practice since it would involve $10^{10}$ regression factors. The answer of the speaker was more or less that it was what they do, with no particular trick but with a large computing power, as if $10^{10}$ was not so big.

Shooting arbitrage - part II

Tue, 05 Jul 2016 09:55:32 +0100

In my previous post, I looked at de-arbitraging volatilities of options of a specific maturity with the shooting method. In reality it is not so practical. While the local volatility will be continuous at the given expiry $T$, it won’t be so at the times $ t \lt T $ because of the interpolation or extrapolation in time. If we consider a single market expiry at time $T$, it is standard practice to extrapolate the implied volatility flatly for $t \lt T$, that is, $w(y,t) = v_T(y) t$ where the variance at time $T$ is defined as $v_T(y)= \frac{1}{T}w(y,T)$. Plugging this into the local variance formula leads to $$\sigma^{\star 2}\left(y, t\right) = \frac{ v_T(y)}{1 - \frac{y}{v_T}\frac{\partial v_T}{\partial y} + \frac{1}{4}\left(-\frac{t^2}{4}-\frac{t}{v_T}+\frac{y^2}{v_T^2}\right)\left(\frac{\partial v_T}{\partial y}\right)^2 + \frac{t}{2}\frac{\partial^{2} v_T}{\partial y^2}}$$ for $t\leq T$. In particular, for $t=0$, we have $$\sigma^{\star 2}\left(y, 0\right) = \frac{ v_T(y)} {1 - \frac{y}{v_T}\frac{\partial v_T}{\partial y} + \frac{1}{4}\left(\frac{y^2}{v_T^2}\right)\left(\frac{\partial v_T}{\partial y}\right)^2}$$ But the first derivative is not continuous, and jumps at $y=y_0$ and $y=y_1$. The local volatility will jump as well around those points. Thus, in practice, the technique can not be used for pricing under local volatility.

Shooting arbitrage - part I

Wed, 22 Jun 2016 09:55:32 +0100

In my previous post, I looked at de-arbitraging volatilities of options of a specific maturity with SVI (re-)calibration. The penalty method can be used beyond SVI. For example I interpolate here with a cubic spline on 11 equidistant nodes the original volatility slice that contains arbitrages and then minimize with Levenberg-Marquardt and the negative local variance denominator penalty on 51 equidistant points. This results in a quite small adjustment to the original volatilities:

Dearbitraging a weak smile on SVI with Damghani's method

Wed, 15 Jun 2016 09:55:32 +0100

Yesterday, I wrote about some calendar spread arbitrages with SVI. Today I am looking at the famous example of butterfly spread arbitrage from Axel Vogt. $$(a, b, m, \rho, \sigma) = (−0.0410, 0.1331, 0.3586, 0.3060, 0.4153)$$ The parameters obey the weak no-arbitrage constraint of Gatheral, and yet produce a negative density, or equivalently, a negative denominator in the local variance Dupire formula. Those parameters are mentioned in Jim Gatheral and Antoine Jacquier paper on arbitrage free SVI volatility surfaces and also in Damghani’s paper dearbitraging a weak smile.

Arbitrage in Zeliade's SVI example

Tue, 14 Jun 2016 09:55:32 +0100

Zeliade wrote an excellent paper about the calibration of the SVI parameterization for the volatility surface in 2008. I just noticed recently that their example calibration actually contained strong calendar spread arbitrages. This is not too surprising if you look at the parameters, they vary wildly between the first and the second expiry.

T	a	b	rho	m	s
0.082	0.027	0.234	0.068	0.100	0.028
0.16	0.030	0.125	-1.0	0.074	0.050
0.26	0.032	0.094	-1.0	0.093	0.041

The calendar spread arbitrage is very visible in total variance versus log-moneyness graph: in those coordinates if lines crosses, there is an arbitrage. This is because the total variance should be increasing with the expiry time.

Arbitrage in Zeliade's example

Dupire Local Volatility with Cash Dividends Part 2

Sun, 29 May 2016 17:01:00 +0200

I had a look at how to price under Local Volatility with Cash dividends in my previous post. I still had a somewhat large error in my FDM price. After too much time, I managed to find the culprit, it was the extrapolation of the prices when applying the jump continuity condition $V(S,t_\alpha^-) = V(S-\alpha, t_\alpha^+) $ for an asset $S$ with a cash dividend of amount $\alpha$ at $ t_\alpha $.

Dupire Local Volatility with Cash Dividends

Thu, 19 May 2016 17:01:00 +0200

The Dupire equation for local volatility has been derived under the assumption of Martingality, that means no dividends or interest rates. The extension to continuous dividend yield is described in many papers or books:

$$\sigma^{\star 2}\left(K, T\right) = \frac{\frac{\partial C}{\partial T}+(r_B-q)K \frac{\partial C}{\partial K} + q C(K,T)}{\frac{1}{2}K^2\frac{\partial^2 C}{\partial K^2}}$$

In this case, $C$ is a regular market Call option price, that can also be obtained from market volatilities via the Black-Scholes formula.

SVI, SABR, or parabola on AAPL?

Thu, 12 May 2016 19:32:42 +0200

In a previous post, I took a look at least squares spline and parabola fits on AAPL 1m options market volatilities. I would have imagined SVI to fit even better since it has 5 parameters, and SABR to do reasonably well.

It turns out that the simple parabola has the lowest RMSE, and SVI is not really better than SABR on that example.

SVI, SABR, least squares parabola fitted to AAPL 1m options

Note that this is just one single example, unlikely to be representative of anything, but I thought this was interesting that in practice, a simple parabola can compare favorably to more complex models.

Adaptive Filon quadrature for stochastic volatility models

Thu, 12 May 2016 19:08:18 +0200

A while ago, I have applied a relatively simple adaptive Filon quadrature to the problem of volatility swap pricing. The Filon quadrature is an old quadrature from 1928 that allows to integrate oscillatory integrand like $f(x)\cos(k x) $ or $f(x)\sin(k x) $. It turns out that combined with an adaptive Simpson like method, it has many advantages over more generic adaptive quadrature methods like Gauss-Lobatto, which is often used on similar problems.

Least Squares Rational Function

Thu, 21 Apr 2016 16:37:24 +0200

In my paper “Fast and Accurate Analytic Basis Point Volatility”, I use a table of Chebyshev polynomials to provide an accurate representation of some function. This is an idea I first saw in the Faddeeva package to represent the cumulative normal distribution with high accuracy, and high performance. It is also simple to find out the Chebyshev polynomials, and which intervals are the most appropriate for those, which makes this technique quite appealing.

Least Squares Spline for Volatility Interpolation

Fri, 19 Feb 2016 18:29:33 +0100

I am experimenting a bit with least squares splines. Existing algorithms (for example from the NSWC Fortran library) usually work with B-splines, a relatively simple explanation of how it works is given in this paper (I think this is how De Boor coded it in the NSWC library). Interestingly there is an equivalent formulation in terms of standard cubic splines, although it seems that the pseudo code on that paper has errors.

The Mystic Parabola

Tue, 16 Feb 2016 22:13:53 +0100

I recently had some fun trying to work directly with the option chain from the Nasdaq website. The data there is quite noisy, but a simple parabola can still give an amazing fit. I will consider the options of maturity two years as illustration. I also relied on a simple implied volatility algorithm that can be summarized in the following steps:

Compute a rough guess for the forward price by using interest, borrow curves and by extrapolating the dividends.
Imply the forward from the European Put-Call parity relationship on the mid prices of the two strikes closes to the rough forward guess. A simple linear interpolation between the two strikes can be used to compute the forward.
Compute the Black implied volatilities as if the option were European using P. Jaeckel algorithm.
Calibrate the proportional dividend amount or the growth rate by minimizing, for example with a Levenberg-Marquardt minimizer, the difference between model and mid-option prices corresponding to the three strikes closest to the forward. The parameters in this case are effectively the dividend amount and the volatilities for Put and Call options (the same volatility is used for both options). The initial guess stems directly from the two previous steps. American option prices are computed by the finite difference method.
Solve numerically the volatilities one by one with the TOMS748 algorithm so that the model prices match the market mid out-of-the-money option prices.

Then I just fit a least squares parabola in variance on log-moneyness, using options trading volumes as weights and obtain the following figure:

Yahoo Finance Implied Volatility

Wed, 03 Feb 2016 16:45:58 +0100

The option chain on Yahoo finance shows an implied volatility number for each call or put option in the last column. I was wondering a bit how they computed that number. I did not exactly find out their methodology, especially since we don’t even know the daycount convention used, but I did find that it was likely just garbage.

A red-herring is for example the large discrepancy between put vols and call vols. For example strike 670, call vol=50%, put vol=32%. This suggests that the two are completely decoupled, and they use some wrong forward (spot price?) to obtain those numbers. If I compute the implied volatilities using put-call parity close to the money to find out the implied forward price, I end up with ask vols of 37% and 34% or call and put mid vols of 33%. By considering the put-call parity, I assume European option prices, which is not correct in this case. It turns out however, that with the low interest rates we live in, there is nearly zero additional value due to the American early exercise.

Is Tufte overrated?

Wed, 03 Feb 2016 16:11:30 +0100

Tufte proposes interesting guidelines to present data, or even to design written semi-scientific papers or books. Some advices are particularly relevant like the careful use of colors (don’t use all the colors of the rainbow just because you can), and in general don’t add lines in a graph or designs that are not directly relevant to the message that needs to be conveyed. There is also a parallel with Feynman message against (Nasa) Powerpoint presentations. But other inspirations, are somewhat doubtful. He seems to have a fetish for old texts. They might be considered pretty, or interesting in some ways, but I don’t find them particularly easy to read. They look more like esoteric books rather than practical books. If you want to write the new Bible for your new cult, it’s probably great, not so sure it’s so great for a more simple subject. Also somewhat surprisingly, his own website is not very well designed, it looks like a maze and very end of 90s.

Linear and Flat forward interpolation with cash dividends

Tue, 19 Jan 2016 09:55:32 +0100

When the dividend curve is built from discrete cash dividends, the dividend yield is discontinuous at the dividend time as the asset price jumps from the dividend amount. This can be particularly problematic for numerical schemes like finite difference methods. In deed, a finite difference grid will make use of the forward yield (eventually adjusted to the discretisation scheme), which explodes then. Typically, if one is not careful about this, then increasing the number of time steps does not increase accuracy anymore, as the spike just becomes bigger on a smaller time interval. A simple work-around is to limit the resolution to one day. This means that intraday, we interpolate the dividend yield.

Moved to hugo

Sun, 20 Dec 2015 21:00:57 +0100

I moved my blog from blogger to Hugo. Blogger really did not evolve since Google take-over in 2003. Wordpress is today much nicer and prettier. It’s clear that Google did not invest at all, possibly because blogs are passé. Compared to mid 2000, there are very few blogs today. Even programming blogs are scarce. It could be interesting to quantify this. My theory is that it is the direct consequence of the popularity of social networks, and especially facebook (possibly also stackoverflow for programmers): people don’t have time anymore to write as their extra-time is used on social networks. Similarly I noticed that almost nobody comments anymore to the point that even Disqus is very rarely used, and again I attribute that to the popularity of sites like reddit. This is why I did not bother with a comment section on my blog, just email me or tweet about it instead.

Controlling the SABR wings with Hagan PDE

Tue, 15 Dec 2015 10:56:00 +0000

On the Wilmott forum, Pat Hagan has recently suggested to cap the equivalent local volatility in order to control the wings and better match CMS prices. It also helps making the SABR approximation better behaved as the expansion is only valid when

$$ 1 + 2\frac{\rho\nu}{\alpha}y(K)+\frac{\nu^2}{\alpha^2}y^2(K) $$

is close to 1.

In the PDE approach (especially the non transformed one), it is very simple, one just needs to update the equivalent local vol as

$$\alpha K^\beta \min\left(M, \sqrt{1 + 2\frac{\rho\nu}{\alpha}y(K)+\frac{\nu^2}{\alpha^2}y^2(K)}\right)$$

While it is straightforward to include in the PDE, it is more difficult to derive a good approximation. The zero-th order behaves as expected, but the first order formula has a unnatural kink, likely because of the non differentiability due to the min function.

The following graphs presents the non capped PDE, the capped PDE with M=4*nu (PDEC4) and M=6*nu (PDEC6) as well as the approximation (Andersen Ratcliffe / Gatheral first order) where I have only taken care of the right wing. The SABR parameters are alpha = 0.0630, beta = 0.7, rho = -0.363, nu = 0.421, T = 10, f = 0.0439.

We can see that the higher the cap is, the closer we are to the standard SABR PDE, and the lower the cap is, the flatter are the wings.

The approximation matches well ATM (it is then equivalent to standard SABR PDE) but then has a discontinuous derivative for the K that reaches the threshold M. Far away, it matches very well again.

Broken Internet?

Mon, 09 Nov 2015 13:40:00 +0000

There is something funny going on with upcoming generic top level domains (gTLDs), they seem to be looked up in a strange manner (at least on latest Linux). For example:

ping chrome

or

ping nexus

returns 127.0.53.53.

While existing official gTLDs don't (ping dental returns "unknown host" as expected). I first thought it was a network misconfiguration, but as I am not the only one to notice this, it's likely a genuine internet issue.

Strange times.

Holiday's read - DFW - Everything and more

Sun, 01 Nov 2015 17:55:00 +0000

I am ambivalent towards David Foster Wallace. He can write the most creative sentences and make innocuous subjects very interesting. At the same time, i never finished his bookInfinite Jest, partly because the characters names are too awkward for me so that i never exactly remember who is who, but also because the story itself is a bit too crazy.

I knew however that a non fiction book on the subject of infinity written by him would make for a very interesting read. And I have not been disappointed. It’s in between maths and philosophy going back to the Greeks up to Gödel through a lot of Cantor following more or less the historical chronology.

Crank-Nicolson and Rannacher Issues with Touch options

Wed, 30 Sep 2015 13:34:00 +0000

I just stumbled upon this particularly illustrative case where the Crank-Nicolson finite difference scheme behaves badly, and the Rannacher smoothing (2-steps backward Euler) is less than ideal: double one touch and double no touch options.

It is particularly evident when the option is sure to be hit, for example when the barriers are narrow, that is our delta should be around zero as well as our gamma. Let's consider a double one touch option with spot=100, upBarrier=101, downBarrier=99.9, vol=20%, T=1 month and a payout of 50K.


Crank-Nicolson shows big spikes in the delta near the boundary

Rannacher shows spikes in the delta as well

Crank-Nicolson spikes are so high that the price is actually a off itself.

The Rannacher smoothing reduces the spikes by 100x but it's still quite high, and would be higher had we placed the spot closer to the boundary. The gamma is worse. Note that we applied the smoothing only at maturity. In reality as the barrier is continuous, the smoothing should really be applied at each step, but then the scheme would be not so different from a simple Backward Euler.

In contrast, with a proper second order finite difference scheme, there is no spike.

Delta with the TR-BDF2 finite difference method - the scale goes from -0.00008 to 0.00008.

Delta with the Lawson-Morris finite difference scheme - the scale goes from -0.00005 to 0.00005

Both TR-BDF2 and Lawson-Morris (based on a local Richardson extrapolation of backward Euler) have a very low delta error, similarly, their gamma is very clean. This is reminiscent of the behavior on American options, but the effect is magnified here.

Clouds

Wed, 02 Sep 2015 15:36:00 +0000

I was wondering how to generate some nice cloudy like texture with a simple program. I first thought about using the Brownian motion, but of course if one uses it raw, with one pixel representing one movement in time, it's just going to look like a very noisy and grainy picture like this:

Normal noise

There is however a nice continuous representation of the Brownian motion : the Paley-Wiener representation

This can produce an interesting smooth pattern, but it is just 1D. In the following picture, I apply it to each row (the column index being time), and then for each column (the row index being time). Of course this produces a symmetric picture, especially as I reused the same random numbers

If I use new random numbers for the columns, it is still symmetric, but destructive rather than constructive.

It turns out that spatial processes are something more complex than I first imagined. It is not a simple as using a N-dimensional Brownian motion, as it would produce a very similar picture as the 1-dimensional one. But this paper has a nice overview of spatial processes. Interestingly they even suggest to generate a Gaussian process using a Precision matrix (inverse of covariance matrix). I never thought about doing such a thing and I am not sure what is the advantage of such a scheme.

There is a standard graphic technique to generate nice textures, originating from Ken Perlin for Disney, it is called simply Perlin Noise. It turns out that several web pages in the top Google results confuse simple Perlin noise with fractal sum of noise that Ken Perlin also helped popularize (see his slides: standard Perlin noise, fractal noise). Those pages also believe that the later is simpler/faster. But there are two issues with fractal sum of noise: the first one is that it relies on an existing noise function - you need to first build one (it can be done with a random number generator and an interpolator), and the second one is that it ends up being more complex to program and likely to evaluate as well, see for example the code needed here. The fractal sum of noise is really a complementary technique.

The insight of Perlin noise is to not generate random color values that would be assigned to shades of grey as in my examples, but to generate random gradients, and interpolate on those gradient in a smooth manner. In computer graphics they like the cosine function to give a little bit of non-linearity in the colors. A good approximation, usually used as a replacement in this context is 3x^2 - 2x^3. It's not much more complicated than that, this web page explains it in great details. It can be programmed in a few lines of code.

very procedural and non-optimized Go code for Perlin noise

Go for Monte-Carlo

Sat, 22 Aug 2015 16:13:00 +0000

I have looked a few months ago already at Julia, Dart, Rust and Scala programming languages to see how practical they could be for a simple Monte-Carlo option pricing.

I forgot the Go language. I had tried it 1 or 2 years ago, and at that time, did not enjoy it too much. Looking at Go 1.5 benchmarks on the computer language shootout, I was surprised that it seemed so close to Java performance now, while having a GC that guarantees pauses of less 10ms and consuming much less memory.

Bumping Correlations

Sat, 25 Jul 2015 18:36:00 +0000

In his book "Monte Carlo Methods in Finance", P. Jäckel explains a simple way to clean up a correlation matrix. When a given correlation matrix is not positive semi-definite, the idea is to do a singular value decomposition (SVD), replace the negative eigenvalues by 0, and renormalize the corresponding eigenvector accordingly.

One of the cited applications is "stress testing and scenario analysis for market risk" or "comparative pricing in order to ascertain the extent of correlation exposure for multi-asset derivatives", saying that "In many of these cases we end up with a matrix that is no longer positive semi-definite".

It turns out that if one bumps an invalid correlation matrix (the input), that is then cleaned up automatically, the effect can be a very different bump. Depending on how familiar you are with SVD, this could be more or less obvious from the procedure,

As a simple illustration I take the matrix representing 3 assets A, B, C with rho_ab = -0.6, rho_ac = rho_bc = -0.5.

   1.00000 -0.60000 -0.50000
-0.60000   1.00000 -0.50000
-0.50000 -0.50000   1.00000

For those rho_ac and rho_bc, the correlation matrix is not positive definite unless rho_ab in in the range (-0.5, 1). One way to verify this is to use the fact that positive definiteness is equivalent to a positive determinant. The determinant will be 1 - 2*0.25 - rho_ab^2 + 2*0.25*rho_ab.

After using P. Jaeckel procedure, we end up with:

   1.00000 -0.56299 -0.46745
-0.56299   1.00000 -0.46745
-0.46745 -0.46745   1.00000

If we bump now rho_bc by 1% (absolute), we end up after cleanup with:

   1.00000 -0.56637 -0.47045
-0.56637   1.00000 -0.46081
-0.47045 -0.46081 1.00000

It turns out that rho_bc has changed by only 0.66% and rho_ac by -0.30%, rho_ab by -0.34%. So our initial bump (0,0,1) has been translated to a bump (-0.34, -0.30, 0.66). In other words, it does not work to compute sensitivities.

One can optimize to obtain the nearest correlation matrix in some norm. Jaeckel proposes a hypersphere decomposition based optimization, using as initial guess the SVD solution. Higham proposed a specific algorithm just for that purpose. It turns out that on this example, they will converge to the same solution (if we use the same norm). I tried out of curiosity to see if that would lead to some improvement. The first matrix becomes

   1.00000 -0.56435 -0.46672
-0.56435   1.00000 -0.46672
-0.46672 -0.46672   1.00000

And the bumped one becomes

   1.00000 -0.56766 -0.46984
-0.56766   1.00000 -0.46002
-0.46984 -0.46002   1.00000

We find back the same issue, rho_bc has changed by only 0.67%, rho_ac by -0.31% and rho_ab by -0.33%. We also see that the SVD correlation or the real near correlation matrix are quite close, as noticed by P. Jaeckel.

Of course, one should apply the bump directly to the cleaned up matrix, in which case it will actually work as expected, unless our bump produces another non positive definite matrix, and then we would have correlation leaking a bit everywhere. It's not entirely clear what kind of meaning the risk figures would have then.

Andreasen Huge extrapolation

Mon, 13 Jul 2015 17:35:00 +0000

There are not many arbitrage free extrapolation schemes. Benaim et al. extrapolation is one of the few that claims it. However, despite the paper’s title, it is not truely arbitrage free. The density might be positive, but the forward is not preserved by the implied density. It can also lead to wings that don’t obey Lee’s moments condition.

On a Wilmott forum, P. Caspers proposed the following counter-example based on extrapolating SABR: $ \alpha=15%, \beta=80%, \nu=50%, \rho=-48%, f=3%, T=20.0 $. He cut this smile at 2.5% and 6% and used the BDK extrapolation scheme with mu=nu=1.

Unintuitive behavior of the Black-Scholes formula - negative volatilities in displaced diffusion extrapolation

Tue, 07 Jul 2015 16:43:00 +0000

I am looking at various extrapolation schemes of the implied volatilities. An interesting one I stumbled upon is due to Kahale. Even if his paper is on interpolation, there is actually a small paragraph on using the same kind of function for extrapolation. His idea is to simply lookup the standard deviation $ \Sigma $ and the forward $f$ corresponding to a given market volatility and slope: $$ c_{f,\Sigma} = f N(d_1) - k N(d_2)$$ with $$ d_1 = \frac{\log(f/k)+\Sigma^2 /2}{\Sigma} $$

Linux Desktops in 2015

Wed, 24 Jun 2015 18:53:00 +0000

I seem to never be entirely happy with any of the linux desktops these days. I have used XFCE on Ubuntu quite a bit in the past year, it mostly works, but I still had minor annoyances:

sometimes (rarely) my laptop would not wake up from sleep.
notifications sometimes keep popping up too much.
on my desktop, experienced strong tearing issues with the Radeon graphic card, except with some very specific combination of video player settings and desktop settings (and then I had annoying redraw issue when pushing volume up/down in movies).

I am satisfied with two different approaches since:

Square Root Crank-Nicolson

Fri, 19 Jun 2015 16:41:00 +0000

C. Reisinger kindly pointed out to me this paper around square root Crank-Nicolson. The idea is to apply a square root of time transformation to the PDE, and discretize the resulting PDE with Crank-Nicolson. Two reasons come to mind to try this:

the square root transform will result in small steps initially, where the solution is potentially not so smooth, making Crank-Nicolson behave better.
it is the natural time of the Brownian motion.

Interestingly, it has nicer properties than what those reasons may suggest. On the Fokker-Planck density PDE, it does not oscillate under some very mild conditions and preserves density positivity at the peak.

Out of curiosity I tried it to price a one touch barrier option. Of course there is an analytical solution in my test case (Black-Scholes assumptions), but as soon as rates are assumed not constant or local volatility is used, there is no other solution than a numerical method. In the later case, finite difference methods are quite good in terms of performance vs accuracy.

The classic Crank-Nicolson gives a reasonable price, but the strong oscillations near the barrier, at every time step are not very comforting.

Crank-Nicolson Prices near the Barrier. Each line is a different time.

Moving to square root of time removes nearly all oscillations on this problem, even with a relatively low number of time steps compared to the number of space steps.

Square Root Crank-Nicolson Prices near the Barrier. Each line is a different time.

We can see that the second step prices are a bit higher than the third step (the lines cross), which looks like a small numerical oscillation in time, even if there is no oscillation is space.

TR-BDF2 Prices near the Barrier. Each line is a different time.

As a comparison, the TR-BDF2 scheme does relatively well: oscillations are removed after the second step, even with the extreme ratio of time steps vs space steps used on this example so that illustrations are clearer - Crank-Nicolson would still oscillate a lot with 10 times less space steps but we would not see oscillation on the square root Crank-Nicolson and a very mild one on TR-BDF2.

The LMG2 scheme (a local richardson extrapolation) does not oscillate at all on this problem but is the slowest:

LMG2 Prices near the Barrier. Each line is a different time.

The square root Crank-Nicolson is quite elegant. It can however not be applied to that many problems in practice, as often some grid times are imposed by the payoff to evaluate, for example in a case of a discrete weekly barrier. But for continuous time problems (density PDE, Vanilla, American, continuous barriers) it's quite good.

In reality, with a continuous barrier, the payoff is not discontinuous at every step, but it is only discontinuous at the first step. So Rannacher smoothing would work very well on that problem:

Rannacher Prices near the Barrier. Each line is a different time.

The somewhat interesting payoff left for the square root Crank-Nicolson is the American.

Decoding Hagan's arbitrage free SABR PDE derivation

Fri, 08 May 2015 16:50:00 +0000

Here are the main steps of Hagan derivation. Let's recall his notation for the SABR model where typically, \$C(F) = F^\beta\$

First, he defines the moments of stochastic volatility:

Then he integrates the Fokker-Planck equation over all A, to obtain

On the backward Komolgorov equation, he applies a Lamperti transform like change of variable:

And then makes another change of variable so that the PDE has the same initial conditions for all moments:

This leads to

It turns out that there is a magical symmetry for k=0 and k=2.

Note that in the second equation, the second derivative applies to the whole. Because of this, he can express \$Q^{(2)}\$ in terms of \$Q^{(0)}\$:

And he plugs that back to the integrated Fokker-Planck equation to obtain the arbitrage free SABR PDE:

There is a simple more common explanation in the world of local stochastic volatility for what's going on. For example, in the particle method paper from Guyon-Labordère, we have the following expression for the true local volatility.

In the first equation, the numerator is simply $Q^{(2)}$ and the denominator $Q^{(0)}$. Of course, the integrated Fokker-Planck equation can be rewritten as:

Matching Hagan PDE SABR with the one-step Andreasen-Huge SABR

Thu, 30 Apr 2015 17:16:00 +0000

I looked nearly two years ago already at the arbitrage free SABR of Andreasen-Huge in comparison to the arbitrage free PDE of Hagan and showed how close the ideas were: Andreasen-Huge relies on the normal Dupire forward PDE using a slightly simpler local vol (no time dependent exponential term) while Hagan works directly on the Fokker-Planck PDE (you can think of it as the Dupire Forward PDE for the density) and uses an expansion of same order as the original SABR formula (which leads to an additional exponential term in the local volatility).

Modern Programming Language for Monte-Carlo

Sat, 18 Apr 2015 22:58:00 +0000

A few recent programming languages sparked my interest:

Julia because of the wide coverage of mathematical functions, and great attention to quality of the implementations. It has also some interesting web interface.
Dart: because it’s a language focused purely on building apps for the web, and has a supposedly good VM.
Rust: it’s the latest fad. It has interesting concepts around concurrency and a focus on being low level all the while being simpler than C.

I decided to see how well suited they would be on a simple Monte-Carlo simulation of a forward start option under the Black model. I am no expert at all in any of the languages, so this is a beginner’s test. I compared the runtime for executing 16K simulations times a multiplier.

Volatility Swap vs Variance Swap Replication - Truncation

Mon, 16 Mar 2015 14:39:00 +0000

I have looked at jump effects on volatility vs. variance swaps. There is a similar behavior on tail events, that is, on truncating the replication. One main problem with discrete replication of variance swaps is the implicit domain truncation, mainly because the variance swap equivalent log payoff is far from being linear in the wings. The equivalent payoff with Carr-Lee for a volatility swap is much more linear in the wings (not so far of a straddle). So we could expect the replication to be less sensitive to the wings truncation. I have done a simple test on flat 40% volatility:

Arbitrage free SABR with negative rates - alternative to shifted SABR

Wed, 11 Mar 2015 18:48:00 +0000

Antonov et al. present an interesting view on SABR with negative rates: instead of relying on a shifted SABR to allow negative rates up to a somewhat arbitrary shift, they modify slightly the SABR model to allow negative rates directly: $$ dF_t = |F_t|^\beta v_t dW_F $$ with \$ v\_t \$ being the standard lognormal volatility process of SABR.

Furthermore they derive a clever semi-analytical approximation for this model, based on low correlation, quite close to the Monte-Carlo prices in their tests. It's however not clear if it is arbitrage-free.

It turns out that it is easy to tweak Hagan SABR PDE approach to this "absolute SABR" model: one just needs to push the boundary \$F\_{min}\$ far away, and to use the absolute value in C(F).

It then reproduces the same behavior as in Antonov et al. paper:

"Absolute SABR" arbitrage free PDE

Antonov et al. graph

I obtain a higher spike, it would look much more like Antonov graph had I used a lower resolution to compute the density: the spike would be smoothed out.

Interestingly, the arbitrage free PDE will also work for high beta (larger than 0.5):

beta = 0.75

It turns out to be then nearly the same as the absorbing SABR, even if prices can cross a little the 0. This is how the bpvols look like with beta = 0.75:

red = absolute SABR, blue = absorbing SABR with beta=0.75

They overlap when the strike is positive.

If we go back to Antonov et al. first example, the bpvols look a bit funny (very symmetric) with beta=0.1:

For beta=0.25 we also reproduce Antonov bpvol graph, but with a lower slope for the left wing:

bpvols with beta = 0.25

It's interesting to see that in this case, the positive strikes bp vols are closer to the normal Hagan analytic approximation (which is not arbitrage free) than to the absorbing PDE solution.

For longer maturities, the results start to be a bit different from Antonov, as Hagan PDE relies on a order 2 approximation only:

absolute SABR PDE with 10y maturity

The right wing is quite similar, except when it goes towards 0, it's not as flat, the left wing is much lower.

Another important aspect is to reproduce Hagan's knee, the atm vols should produce a knee like curve, as different studies show (see for example this recent Hull & White study or this other recent analysis by DeGuillaume). Using the same parameters as Hagan (beta=0, rho=0) leads to a nearly flat bpvol: no knee for the absolute SABR, curiously there is a bump at zero, possibly due to numerical difficulty with the spike in the density:

The problem is still there with beta = 0.1:

Overall, the idea of extending SABR to the full real line with the absolute value looks particularly simple, but it's not clear that it makes real financial sense.

Variance swaps on a foreign asset

Tue, 24 Feb 2015 13:50:00 +0000

There is very little information on variance swaps on a foreign asset. There can be two kinds of contracts:

one that pays the foreign variance in a domestic currency, this is a quanto contract as the exchange rate is implicitly fixed.
one that pays the foreign variance, multiplied by the fx rate at maturity. This is a flexo contract, and is just about buying a variance swap from a foreign bank. The price of such a contract today is very simple, just the standard variance swap price multiplied by the fx rate today (change of measure).

For quanto contracts, it's not so obvious a priori. If we consider a stochastic volatility model for the asset, the replication formula will not be applicable directly as the stochastic volatility will appear in the quanto drift correction. Furthermore, vanilla quanto option prices can not be computed simply as under Black-Scholes, a knowledge of the underlying model is necessary.

Interestingly, under the Schobel-Zhu model, it is simple to fit an analytic formula for the quanto variance swap. The standard variance swap price is:

The quanto variance swap can be priced with the same formula using a slightly different theta:

We can use it to assess the accuracy of a naive quanto option replication where we use the ATM quanto forward instead of the forward in the variance swap replication formula.

Interestingly, the quanto forward approximation turns out to be very accurate and the correction is important. The price without correction is the price with zero correlation, and we see it can be +/-5% off in this case.

The local vol price seems a bit off, I am not sure exactly why. It could be due the discretization, the theoretical variance should be divided by (N-1) but here we divide by N where N is the number of observations. That would still lead to a skewed price but better centered around correlation 0.

It's also a bit surprising that local vol is worse than the simpler ATM quanto forward approximation: it seems that it's extracting the wrong information to do a more precise quanto correction, likely related to the shift of stochastic volatility under the domestic measure.

Jumps impact: Variance swap vs volatility swap

Fri, 20 Feb 2015 13:24:00 +0000

Beside the problem with the discreteness of the replication, variance swaps are sensitive to jumps. This is an often mentioned reason for the collapse of the single name variance swap market in 2008 as jumps are more likely on single name equities.

Those graphs are the result of Monte-Carlo simulations with various jump sizes using the Bates model, and using Local Volatility implied from the Bates vanilla prices. The local volatility price will be the same price as per static replication for the variance swap, and we can see it they converge when there is no jump.

Variance Swap Replication : Discrete or Continuous?

Thu, 19 Feb 2015 18:45:00 +0000

People regularly believe that Variance swaps need to be priced by discrete replication, because the market trades only a discrete set of options.

In reality, a discrete replication will misrepresent the tail, and can be quite arbitrary. It looks like the discrete replication as described in Derman Goldman Sachs paper is in everybody's mind, probably because it's easy to grasp. Strangely, it looks like most forget the section "Practical problems with replication" on p27 of his paper, where you can understand that discrete replication is not all that practical.

Reflecting on all of this, I noticed it was possible to create more accurate discrete replications easily, and that those can have vastly different hedging weights. It is a much better idea to just replicate the log payoff continuously with a decent model for interpolation and extrapolation and imply the hedge from the greeks.

I wrote a small paper around this here.

GTK 3.0 / Gnome 3.0 annoyance

Sun, 08 Feb 2015 22:30:00 +0000

It’s quite incredible that Gnome 3.0 was almost an identical mess as KDE 4.0 had been a year or two earlier. Both are much better now, more stable, but both also still have their issues, and don’t feel like a real improvement over Gnome 2.0 or KDE 3.5.

Now the main file manager for Gnome 3.0, Nautilus has buttons with nearly identical icons that mean vastly different things, one is a menu, the other is a list view. Also it does not integrate with other desktops well from a look and feel perpective, here is a screenshot under XFCE (KDE would not look better).The push for window buttons inside the toolbar makes for a funny looking window. In Gnome Shell, it’s not much better, plus there are some windows with a dark theme and some with a standard theme all mixed together.

Monte Carlo & Inverse Cumulative Normal Distribution

Tue, 03 Feb 2015 14:53:00 +0000

In most financial Monte-Carlo simulations, there is the need of generating normally distributed random numbers. One technique is to use the inverse cumulative normal distribution function on uniform random numbers. There are several different popular numerical implementations:

Wichura AS241 (1988)
Moro "The full Monte" (1995)
Acklam (2004)
Shaw breakless formula optimized for GPUs (2011)

W. Shaw has an excellent overview of the accuracy of the various methods in his paper Refinement of the normal quantile.

But what about performance? In Monte-Carlo, we could accept a slighly lower accuracy for an increase in performance.

I tested the various methods on the Euler full truncation scheme for Heston using a small timestep (0.01). Here are the results with Sobol quasi-rng:

AS241 0.9186256922511046 0.42s
MORO 0.9186256922459066 0.38s
ACKLAM 0.9186256922549364 0.40s
ACKLAM REFINED 0.9186256922511045 2.57s
SHAW-HYBRID 0.9186256922511048 0.68s

In practice, the most accurate algorithm, AS241, is of comparable speed as the newer but less accurate algorithms of MORO and ACKLAM. Acklam refinement to go to double precision (which AS241 is) kills its performance.

What about the Ziggurat on pseudo rng only? Here are the results with Mersenne-Twister-64, and using the Doornik implementation of the Ziggurat algorithm:

AS241 0.9231388565879476 0.49s
ZIGNOR 0.9321405648313437 0.44s

There is a more optimized algorithm, VIZIGNOR, also from Doornik which should be a bit faster. As expected, the accuracy is quite lower than with Sobol, and the Ziggurat looks worse. This is easily visible if one plots the implied volatilities as a function of the spot for AS241 and for ZIGNOR.

AS241 implied volatility on Mersenne-Twister

ZIGNOR implied volatility on Mersenne-Twister

Zignor is much noisier.

Note the slight bump in the scheme EULER-FT-BK that appears because the scheme, that approximates the Broadie-Kaya integrals with a trapeze (as in Andersen QE paper), does not respect martingality that well compared to the standard full truncated Euler scheme EULER-FT, and the slightly improved EULER-FT-MID where the variance integrals are approximated by a trapeze as in Van Haastrecht paper on Schobel-Zhu:

This allows to leak less correlation than the standard full truncated Euler.

Local Stochastic Volatility - Particles and Bins

Fri, 30 Jan 2015 12:03:00 +0000

In an earlier post, I mentioned the similarities between the Guyon-Labordere particle method and the Vanderstoep-Grzelak-Oosterlee “bin” method to calibrate and price under Local Stochastic volatility. I will be a bit more precise here. The same thing, really

The particle method can be seen as a generalization of the “bin” method. In deed, the bin method consists in doing the particle method using a histogram estimation of the conditional variance. The histogram estimation can be more or less seen as a very basic rectangle kernel with the appropriate bandwidth. The “bin” method is then just the particle method with another kernel (wiki link) (in the particle method, the kernel is a quartic with bandwidth defined by some slightly elaborate formula). A very good paper on this is Silverman Density estimation for statistics and data analysis, referenced by Guyon-Labordere.

Flat Volatility Surfaces & Discrete Dividends

Tue, 25 Nov 2014 13:58:00 +0000

In papers around volatility and cash (discrete) dividends, we often encounter the example of the flat volatility surface. For example, the OpenGamma paper presents this graph:

It shows that if the Black volatility surface is fully flat, there are jumps in the pure volatility surface (corresponding to a process that includes discrete dividends in a consistent manner) at the dividend dates or equivalently if the pure volatility surface is flat, the Black volatility jumps.

This can be traced to the fact that the Black formula does not respect C(S,K,Td-) = C(S,K-d,Td) as the forward drops from F(Td-) to F(Td-)-d where d is dividend amount at td, the dividend ex date.

Unfortunately, those examples are not very helpful. In practice, the market observables are just Black volatility points, which can be interpolated to volatility slices for each expiry without regards to dividends, not a full volatility surface. Discrete dividends will mostly happen between two slices: the Black volatility jump will happen on some time-interpolated data.

While the jump size is known (it must obey to the call price continuity), the question of how one should interpolate that data until the jump is far from trivial even using two flat Black volatility slices.

The most logical is to consider a model that includes discrete dividends consistently. For example, one can fully lookup the Black volatility corresponding the price of an option assuming a piecewise lognormal process with jumps at the dividend dates. It can be priced by applying a finite difference method on the PDE. Alternatively, Bos & Vandermark propose a simple spot and strike adjusted Black formula that obey the continuity requirement (the Lehman model), which, in practice, stays quite close to the piecewise lognormal model price. Another possibility is to rely on a forward modelling of the dividends, as in Buehler (if one is comfortable with the idea that the option price will then depend ultimately on dividends past the option expiry).

Recently, a Wilmott article suggested to only rely on the jump adjustment, but did not really mention how to find the volatility just before or just after the dividend. Here is an illustration of how those assumptions can change the volatility in between slices using two dividends at T=0.9 and T=1.1.

In the first graph, we just interpolate linearly in forward moneyness the pure vol from the Bos & Vandermark formula, as it should be continuous with the forward (the PDE would give nearly the same result) and compute the equivalent Black volatility (and thus the jump at the dividend dates).

In the second graph, we interpolate linearly the two Black slices, until we find a dividend, at which point we impose the jump condition and repeat the process until the next slice. We process forward (while the Wilmott article processes backward) as it seemed a bit more natural to make the interpolation not depend on future dividends. Processing backward would just make the last part flat and first part down-slopping. On this example backward would be closer to the Bos Black volatility, but when the dividends are near the first slice, the opposite becomes true.

While the scale of those changes is not that large on the example considered, the choice can make quite a difference in the price of structures that depend on the volatility in between slices. A recent example I encountered is the variance swap when one includes adjustment for discrete dividends (then the prices just after the dividend date are used).

To conclude, if one wants to use the classic Black formula everywhere, the volatility must jump at the dividend dates. Interpolation in time is then not straightforward and one will need to rely on a consistent model to interpolate. It is not exactly clear then why would anyone stay with the Black formula except familiarity.

Machine Learning & Quantitative Finance

Tue, 18 Nov 2014 12:34:00 +0000

There is an interesting course on Machine Learning on Coursera, it does not require much knowledge and yet manages to teach quite a lot.

I was struck by the fact that most techniques and ideas apply also to problems in quantitative finance.

Linear regression: used for example in the Longstaff-Schwartz approach to price Bermudan options with Monte-Carlo. Interestingly the teacher insists on feature normalization, something we can forget easily, especially with the polynomial features.
Gradient descent: one of the most basic minimizer and we use minimizers all the time for model calibration.
Regularization: in finance, this is sometimes used to smooth out the volatility surface, or can be useful to add stability in calibration. The lessons are very practical, they explain well how to find the right value of the regularization parameter.
Neural networks: calibrating a model is very much like training a neural network. The backpropagation is the same thing as the adjoint differentiation. It’s very interesting to see that it is a key feature for Neural networks, otherwise training would be much too slow and Neural networks would not be practical. Once the network is trained, it is evaluated relatively quickly forward. It’s basically the same thing as calibration and then pricing.
Support vector machines: A gaussian kernel is often used to represent the frontier. We find the same idea in the particle Monte-Carlo method.
Principal component analysis: can be applied to the covariance matrix square root in Monte-Carlo simulations, or to “compress” large baskets, as well as for portfolio risk.

It’s also interesting to hear the teacher repeating that people should not try possible improvements at random (often because they have only one idea) but analyze before what makes the most sense. And that can imply digging in the details, looking at what’s going on 100 samples.

Pseudo-Random vs Quasi-Random Numbers

Wed, 12 Nov 2014 17:05:00 +0000

Quasi-Random numbers (like Sobol) are a relatively popular way in finance to improve the Monte-Carlo convergence compared to more classic Pseudo-Random numbers (like Mersenne-Twister). Behind the scenes one has to be a bit more careful about the dimension of the problem as the Quasi-Random numbers depends on the dimension (defined by how many random variables are independent from each other).

For a long time, Sobol was limited to 40 dimensions using the so called Bratley-Fox direction numbers (his paper actually gives the numbers for 50 dimensions). Later Lemieux gave direction numbers for up to 360 dimensions. Then, P. Jäckel proposed some extension with a random initialization of the direction vectors in his book from 2006. And finally Joe & Kuo published direction numbers for up to 21200 dimensions.

But there are very few studies about how good are real world simulations with so many quasi-random dimensions. A recent paper "Fast Ninomiya-Victoir Calibration of the Double-Mean-Reverting Model" by Bayer, Gatheral & Karlsmark tests this for once, and the results are not so pretty:

With their model, the convergence with Sobol numbers becomes worse when the number of time-steps increases, that is when the number of dimension increases. There seems to be even a threshold around 100 time steps (=300 dimensions for Euler) beyond which a much higher number of paths (2^13) is necessary to restore a proper convergence. And they use the latest and greatest Joe-Kuo direction numbers.

Still the total number of paths is not that high compared to what I am usually using (2^13 = 8192). It's an interesting aspect of their paper: the calibration with a low number of paths.

Integrating an oscillatory function

Wed, 05 Nov 2014 16:48:00 +0000

Recently, some instabilities were noticed in the Carr-Lee seasoned volatility swap price in some situations.

The Carr-Lee seasoned volatility swap price involve the computation of a double integral. The inner integral is really the problematic one as the integrand can be highly oscillating.

I first found a somewhat stable behavior using a specific adaptive Gauss-Lobatto implementation (the one from Espelid) and a change of variable. But it was not very satisfying to see that the outer integral was stable only with another specific adaptive Gauss-Lobatto (the one from Gander & Gauschi, present in Quantlib). I tried various choices of adaptive (coteda, modsim, adaptsim,...) or brute force trapezoidal integration, but either they were order of magnitudes slower or unstable in some cases. Just using the same Gauss-Lobatto implementation for both would fail...

I then noticed you could write the integral as a Fourier transform as well, allowing the use of FFT. Unfortunately, while this worked, it turned out to require a very large number of points for a reasonable accuracy. This, plus the tricky part of defining the proper step size, makes the method not so practical.

I had heard before of the Filon quadrature, which I thought was more of a curiosity. The main idea is to integrate exactly x^n * cos(k*x). One then relies on a piecewise parabolic approximation of the function f to integrate f(x) * cos(k*x). Interestingly, a very similar idea has been used in the Sali quadrature method for option pricing, except one integrates exactly x^n * exp(-k*x^2).

It turned out to be remarkable on that problem, combined with a simple adaptive Simpson like method to find the right discretization. Then as if by magic, any outer integration quadrature worked.

The elusive reference: the Lamperti transform

Mon, 03 Nov 2014 11:23:00 +0000

Without knowing that it was a well known general concept, I first noticed the use of the Lamperti transform in the Andersen-Piterbarg “Interest rate modeling” book p.292 “finite difference solutions for general phi”.

Pat Hagan used that transformation for a better discretization of the arbitrage free SABR PDE model.I then started to notice the use of this transformation in many more papers. The first one I saw naming it “Lamperti transform” was the paper from Ait-Sahalia Maximum likelyhood estimation of discretely sampled diffusions: a closed-form approximation approach. Recently those closed form formulae have been applied to the quadrature method (where one integrates the transition density by a quadrature rule) in “Advancing the universality of quadrature methods to any underlying process for option pricing”. There is also a recent interesting application to Monte-Carlo simulation in “Unbiased Estimation with Square Root Convergence for SDE Models”.

Barrier options under negative rates: complex numbers to the rescue

Thu, 02 Oct 2014 11:58:00 +0000

I stumbled upon an unexpected problem: the one touch barrier formula can break down under negative rates. While negative rates can sound fancy, they are actually quite real on some markets. Combined with relatively low volatilities, this makes the standard Black-Scholes one touch barrier formula blow up because somewhere the square root of a negative number is taken.

At first, I had the idea to just floor the number to 0. But then I needed to see if this rough approximation would be acceptable or not. So I relied on a TR-BDF2 discretization of the Black-Scholes PDE, where negative rates are not a problem.

Later, I was convinced that we ought to be able to find a closed form formula for the case of negative rates. I went back to the derivation of the formula, the book from Kwok is quite good on that. The closed form formula just stems from being the solution of an integral of the first passage time density (which is a simpler way to compute the one touch price than the PDE approach). It turns out that, then, the closed form solution to this integral with negative rates is just the same formula with complex numbers (there are actually some simplifications then).

It is a bit uncommon to use the cumulative normal distribution on complex numbers, but the error function on complex numbers is more popular: it's actually even on the wikipedia page of the error function. And it can be computed very quickly with machine precision thanks to the Faddeeva library.

With this simple closed form formula, there is no need anymore for an approximation. I wrote a small paper around this here.

Later a collegue made the remark that it could be interesting to have the bivariate complex normal distribution for the case of partial start one touch options or partial barrier option rebates (not sure if those are common). Unfortunately I could not find any code or paper for this. And after asking Prof. Genz (who found a very elegant and fast algorithm for the bivariate normal distribution), it looks like an open problem.

Initial Guesses for SVI - A Summary

Fri, 26 Sep 2014 10:46:00 +0000

I have been looking at various ways of finding initial guesses for SVI calibration (Another SVI Initial Guess, More SVI Initial Guesses, SVI and long maturities issues). I decided to write a paper summarizing this. I find that the process of writing a paper makes me think more carefully about a problem.

In this case, it turns out that the Vogt initial guess method (guess via asymptotes and minimum variance) is actually very good as long as one has a good way to lookup the asymptotes (the data is not always convex, while SVI is) and as long as rho is not close to -1, that is for long maturity affine like smiles, where SVI is actually more difficult to calibrate properly due to the over-parameterisation in those cases.

Asymptotic Behavior of SVI vs SABR

Tue, 23 Sep 2014 12:06:00 +0000

The variance under SVI becomes linear when the log-moneyness is very large in absolute terms. The lognormal SABR formula with beta=0 or beta=1 has a very different behavior. Of course, the theoretical SABR model has actually a different asymptotic behavior.

As an illustration, we calibrate SABR (with two different values of beta) and SVI against the same implied volatility slice and look at the wings behavior.

While the Lee moments formula implies that the variance should be at most linear, something that the SABR formula does not respect. It is in practice not the problem with SABR as the actual Lee boundary: V(x) < 2|x|/T (where V is the square of the implied volatility and x the log-moneyness) is attained for extremely low strikes only with SABR, except maybe for very long maturities.

A related behavior is the fact that the lognormal SABR formula can actually match steeper curvatures at the money than SVI for given asymptotes.

SVI and long maturities issues

Fri, 01 Aug 2014 12:51:00 +0000

On long maturities equity options, the smile is usually very much like a skew: very little curvature. This usually means that the SVI rho will be very close to -1, in a similar fashion as what can happen for the the correlation parameter of a real stochastic volatility model (Heston, SABR).

In terms of initial guess, I looked at the more usual use cases and showed that matching a parabola at the minimum variance point often leads to a decent initial guess if one has an ok estimate of the wings. We will see here that we can do also something a bit better than just a flat slice at-the-money in the case where rho is close to -1.

More SVI Initial Guesses

Thu, 31 Jul 2014 14:54:00 +0000

In the previous post, I showed one could extract the SVI parameters from a best fit parabola at-the-money. It seemed to work reasonably well, but I found some real market data where it can be much less satisfying.

Sometimes (actually not so rarely) the ATM slope and curvatures can't be matched given rho and b found through the asymptotes. As a result if I force to just match the curvature and set m=0 (when the slope can't be matched), the simple ATM parabolic guess looks shifted. It can be much worse than this specific example.

It is then a bit clearer why Vogt looked to match the lowest variance instead of ATM. We can actually also fit a parabola at the lowest variance (MV suffix in the graph) instead of ATM. It seems to fit generally better.

I also tried to estimate the asymptotic slopes more precisely (using the slope of the 5-points parabola at each end), but it seems to not always be an improvement.

However this does not work when rho is close to -1 or 1 as there is then no minimum. Often, matching ATM also does not work when rho is -1 or 1. This specific case, but quite common as well for longer expiries in equities need more thoughts, usually a constant slice is ok, but this is clearly where Zeliade's quasi explicit method shines.

So far it still all looks good, but then looking at medium maturities (1 year), sometimes all initial guesses don't look comforting (although Levenberg-Marquardt minimization still works on those - but one can easily imagine that it can break as well, for example by tweaking slightly the rho/b and look at what happens then).

There is lots of data on this 1 year example. One can clearly see the problem when the slope can not be fitted ATM (SimpleParabolicATM-guess), and even if by chance when it can (TripleParabolicATM-guess), it's not so great.
Similarly fitting the lowest variance leads only to a good fit of the right wing and a bad fit everywhere else.

Still, as if by miracle, everything converges to the best fit on this example (again one can find cases where some guesses don't converge to the best fit). I have added some weights +-20% around the money, to ensure that we capture the ATM behavior accurately (otherwise the best fit is funny).

It is interesting to see that if one minimizes the min square sum of variances (what I do in Vogt-LM, it's in theory slightly faster as there is no sqrt function cost) it results in an ugly looking steeper curvature, while if we just minimize the min square sum of volatilities (what I do in SimpleParabolicMV_LM), it looks better.

Another SVI Initial Guess

Tue, 29 Jul 2014 14:39:00 +0000

The SVI formula is:
$$w(k) = a + b ( \rho (k-m) + \sqrt{(k-m)^2+ \sigma^2}$$
where k is the log-moneyness, w(k) the implied variance at a given moneyness and a,b,rho,m,sigma the 5 SVI parameters.

A. Vogt described a particularly simple way to find an initial guess to fit SVI to an implied volatility slice a while ago. The idea to compute rho and sigma from the left and right asymptotic slopes. a,m are recovered from the crossing point of the asymptotes and sigma using the minimum variance.

Later, Zeliade has shown a very nice reduction of the problem to 2 variables, while the remaining 3 can be deduced explicitly. The practical side is that constraints are automatically included, the less practical side is the choice of minimizer for the two variables (Nelder-Mead) and of initial guess (a few random points).

Instead, a simple alternative is the following: given b and rho from the asymptotic slopes, one could also just fit a parabola at-the-money, in a similar spirit as the explicit SABR calibration, and recover explicitly a, m and sigma.

To illustrate I take the data from Zeliade, where the input is already some SVI fit to market data.

3M expiry - Zeliade data

4Y expiry, Zeliade data

We clearly see that ATM the fit is better for the parabolic initial guess than for Vogt, but as one goes further away from ATM, Vogt guess seems better.

Compared to SABR, the parabola itself fits decently only very close to ATM. If one computes the higher order Taylor expansion of SVI around k=0, powers of (k/sigma) appear, while sigma is often relatively small especially for short expiries: the fourth derivative will quickly make a difference.

On implied volatilities stemming from a SABR fit of the SP500, here is how the various methods behave:

1M expiry on SABR data

4Y expiry on SABR data

As expected, because SABR (and thus the input implied vol) is much closer to a parabola, the parabolic initial guess is much better than Vogt. The initial guess of Vogt is particularly bad on long expiries, although it will still converge quite quickly to the true minimum with Levenberg-Marquardt.

In practice, I have found the method of Zeliade to be very robust, even if a bit slower than Vogt, while Vogt can sometimes (rarely) be too sensitive to the estimate of the asymptotes.

The parabolic guess method could also be applied to always fit exactly ATM vol, slope and curvature, and calibrate rho, b to gives the best overall fit. It might be an idea for the next blog post.

New SABR Formulae

Wed, 16 Jul 2014 22:35:00 +0000

In a talk at the Global Derivatives conference of Amsterdam (2014), Pat Hagan presented some new SABR formulas, supposedly close to the arbitrage free PDE behavior.

I tried to code those from the slides, but somehow that did not work out well on his example, I just had something very close to the good old SABR formulas. I am not 100% sure (only 99%) that it is due to a mistake in my code. Here is what I was looking to reproduce:

Pat Hagan Global Derivatives example

Fortunately, I then found in some thesis the idea of using Andersen & Brotherton-Ratcliffe local volatility expansion. In deed, the arbitrage free PDE from Hagan is equivalent to some Dupire local volatility forward PDE (see http://papers.ssrn.com/abstract=2402001), so Hagan just gave us the local volatility expansion to expand on (the thesis uses Doust, which is not so different in this case).

And then it produces on this global derivatives example the following:

The AB suffix are the new SABR formula. Even though the formulas are different, that looks very much like Hagan's own illustration (with a better scale)!

I have a draft paper around this and more practical ideas to calibrate SABR:
http://papers.ssrn.com/abstract=2467231

Heston or Schobel-Zhu issues with short expiries

Thu, 03 Jul 2014 23:28:00 +0000

It's relatively well known that Heston does not fit the market for short expiries. Given that there are just 5 parameters to fit a full surface, it's almost logical that one part of the surface of it is not going to fit well the market.
I was more surprised to see how bad Heston or Schobel-Zhu were to fit a single short expiry volatility slice. As an example I looked at SP500 options with 1 week expiry. It does not really matter if one forces kappa and rho to constant values (even to 0) the behavior is the same and the error in fit does not change much.

Schobel-Zhu fit for a slice of maturity 1 week

In this graph, the brown, green and red smiles corresponds to Schobel-Zhu fit using an explicit guess (matching skew & curvature ATM), using Levenberg-Marquardt on this guess, and using plain differential evolution.
What happens is that the smiles flattens to quickly in the strike dimension. One consequence is that the implied volatility can not be computed for extreme strikes: the smile being too low, the price becomes extremely small, under machine epsilon and the numerical method (Cos) fails. There is also a bogus angle in the right wing, because of numerical error. I paid attention to ignore too small prices in the calibration by truncating the initial data.

Heston fit, with Lord-Kahl (exact wings)

SABR behaves much better (fixing beta=1 in this case) in comparison (As I use the same truncation as for Schobel-Zhu, the flat left wing part is ignored).

SABR fit for a slice of maturity 1 week

For longer expiries, Heston & Schobel-Zhu, even limited to 3 parameters, actually give a better fit in general than SABR.

On the Role of Static Types and Generic Types on Productivity

Sun, 29 Jun 2014 10:40:00 +0000

Most developers have strong opinions on dynamic types programming languages vs static types programming languages. The former is often assumed to be good for small projects/prototyping while the later better for bigger projects. But there is a surprisingly small number of studies to back those claims.

One such study is "An experiment about static and dynamic type systems: doubts about the positive impact of static type systems on development time" and came to the conclusion that on a small project, static typing did not decrease programming time, and actually increased debugging time. However 4 years later, "An empirical comparison of static and dynamic type systems on API usage in the presence of an IDE: Java vs. groovy with eclipse" shows that a developer is 2x more productive with Java than with Groovy using an unknown API. This contrasts a bit (but does not contradict) with their previous study "Static Type Systems (Sometimes) have a Positive Impact on the Usability of Undocumented Software: An Empirical Evaluation" that showed Groovy to be more productive on small projects. One problem is that all these studies stem from the same person.

It's more interesting to look at generic types vs raw types use, where even less studies have been done. "Do developers benefit from generic types?: an empirical comparison of generic and raw types in java" concludes that generic types do not provide any advantages to fix typing errors, hardly surprising in my opinion. Generic types (especially with type erasure as in Java) is the typical idea that sounds good but that in practice does not really help: it makes the code actually more awkward to read and tend to make developers too lazy to create new classes that would often be more appropriate than a generic type (think Map<String,List<Map<String, Date>>>).

Moore-Penrose Inverse & Gauss-Newton SABR Minimization

Tue, 24 Jun 2014 15:29:00 +0000

I have found a particularly nice initial guess to calibrate SABR. As it is quite close to the true best fit, it is tempting to use a very simple minimizer to go to the best fit. Levenberg-Marquardt works well on this problem, but can we shave off a few iterations?

I firstly considered the basic Newton's method, but for least squares minimization, the Hessian (second derivatives) is needed. It's possible to obtain it, even analytically with SABR, but it's quite annoying to derive it and code it without some automatic differentiation tool. It turns out that as I experimented with the numerical Hessian, I noticed that it actually did not help convergence in our problem. Gauss-Newton converges similarly (likely because the initial guess is good), and what's great about it is that you just need the Jacobian (first derivatives). Here is a good overview of Newton, Gauss-Newton and Levenberg-Marquardt methods.

While Gauss-Newton worked on many input data, I noticed it failed also on some long maturities equity smiles. The full Newton's method did not fare better. I had to take a close look at the matrices involved to understand what was going on. It turns out that sometimes, mostly when the SABR rho parameter is close to -1, the Jacobian would be nearly rank deficient (a row close to 0), but not exactly rank deficient. So everything would appear to work, but it actually misbehaves badly.

My first idea was to solve the reduced problem if a row of the Jacobian is too small, by just removing that row, and keep the previous value for the guess corresponding to that row. And this simplistic approach made the process work on all my input data. Here is the difference in RMSE compared to a highly accurate Levenberg-Marquardt minimization for 10 iterations:

Later, while reading some more material related to least square optimization, I noticed the use of the Moore-Penrose inverse in cases where a matrix is rank deficient. The Moore-Penrose inverse is defined as:
$$ M^\star = V S^\star U^T$$
where $ S^\star $ is the diagonal matrix with inverted eigenvalues and 0 if those are deemed numerically close to 0, and $U, V$ the eigenvectors of the SVD decomposition:
$$M=U S V^T$$
It turns out to work very well, beside being simpler to code, I expected it to be more or less equivalent to the previous approach (a tiny bit slower but we don't care as we deal with small matrices, and the real slow part is the computation of the objective function and the Hessian, which is why looking at iterations is more important).

It seems to converge a little bit less quickly, likely due to the threshold criteria that I picked (1E-15).
Three iterations is actually most of the time (90%) more than enough to achieve a good accuracy (the absolute RMSE is between 1E-4 and 5E-2) as the following graph shows. The few spikes near 1E-3 represent too large errors, the rest is accurate enough compared to the absolute RMSE.

To conclude, we have seen that using the Moore-Penrose inverse in a Gauss-Newton iteration allowed the Gauss-Newton method to work on rank-deficient systems.
I am not sure how general that is, in my example, the true minimum either lies inside the region of interest, or on the border, where the system becomes deficient. Of course, this is related to a "physical" constraint, here namely rho > -1.

One Interview Question for Job Seekers in Finance

Thu, 19 Jun 2014 21:51:00 +0000

I presented in an earlier post that I was mostly disillusioned with interview questions, it’s better to find out if you can learn something out of a candidate.Well there is maybe one very simple question that could be revealing, for people who pretend to be vaguely familiar with Black-Scholes:What is the price of an at-the-money binary option under very high volatility? Alternatively it can be asked with just an at-the-money european option under very high volatility.What makes think of it is that some “product manager” recently tested risk with volatilities at 300% and was wondering why they did not see any vega (based on a 1% additive shift), and opened bugs, generated noise…

On the importance of accuracy for bpvol solvers

Thu, 12 Jun 2014 17:31:00 +0000

While I was playing around calibrating the arbitrage free SABR model from Hagan (using the PDE on probability density approach), I noticed a misbehavior for some short maturity smiles. I thought it was due to the PDE implementation. Actually some of it was, but the remaining large error was due to the bpvol solver.

I initially took the same approach as Choi et al. in my solver, that is to work with in-the-money prices (they work with straddles) because it’s nice and convenient. I thought it was no big deal if prices lower than 1E-16 were not solved. It turns out I was wrong. Choi et al. solver has the same issue.

throws Exception

Tue, 27 May 2014 10:49:00 +0000

There was a big debate at work around Exception declaration in a Java API. I was quite surprised that such an apparently simple subject could end up being so controversial. The controversy was around the choice of declaring in the interfaces:

void myMethod() throws Exception

instead of

void myMethod() throws MyAPIException
void myMethod() throws MyAPIRuntimeException
void myMethod()

where MyAPI represents either a generic API related exception or a specific exception related to the method in question.

The choice of "throws Exception" did not even occur to me as a possibility, but after some digging, I found that some relatively famous libraries actually followed that principle at one point, for example Apache Struts 1.x or Spring MVC.

More modern libraries, like Google Guava, commons-math 3.x, Struts 2.x generally favor MyAPIRuntimeException where MyAPI is actually context-specific. Some old popular libraries declare a checked Exception, for example the HibernateException in Hibernate.

This seems to be a recurring subject on Stackoverflow:
Stackoverflow - Java interface throws Exception best practice
Stackoverflow - What to put in the throws clause of an interface method

But those are quite poor in terms of explanations. The best comments on this subjects are from:
Anders Hejlsberg (C#, Delphi, Turbo Pascal creator) - The Trouble with Checked Exceptions
James Gosling (Java creator) - Failure and Exceptions

This comment from Anders is particularly acute:
"To work around this requirement, people do ridiculous things. For example, they decorate every method with, "throws Exception." That just completely defeats the feature, and you just made the programmer write more gobbledy gunk. That doesn't help anybody. "

Today I believe the API in question declares "throws Exception"...

KDE, XFCE, Gnome-Shell in 2014

Sun, 25 May 2014 09:26:00 +0000

Many people (and notoriously, Linus Torvald) complained about Gnome-shell, especially the early iterations. Similarly KDE 4 was a nightmare of instability and inflexibility when it came out. And XFCE has always sounded a bit too basic. the moves of Gnome and KDE were particularly shocking as the earlier iteration: Gnome 2 and KDE 3 were well appreciated, productive environments.

Gnome Shell 3.10

It took me a bit of time to get used to it, and in the early stages I went to KDE 4 for a while, only to come back to it later.

Positive aspects: lots of space on the desktop, things don't get in the way, looks good,very good desktop overview (fast and well presented), a dock by default, great external monitor support (plug and play, remembers settings automatically), best OSD (volume) of all.
Negative aspects: the notifications bar looks awkward and badly integrated (better with an extension), still unstable and big memory leaks (on Fedora 20, where the integration should be the best, it regularly crashes, starts with 300Mb and goes up to 1Gb in a couple of days), fallback-session completely useless as one can not customize it at all. But the killer for my work was inability to share the desktop with Webex, while XFCE could.

KDE

I gave it a long try especially in 2012, it has not changed much in 2014. My opinion of it fell when I tried it a very short time after months of Gnome Shell, and even more so after seeing the trouble my parents had with it, compared to Gnome 2.

Positive aspects: desktop search (needs to be configured in order to scan only the relevant folders, used to be slow and resource intensive, not so much in 2014)
Negative aspects: resource hog, awful start menu, too many shiny effects by default that only distract the user from his task, silly concepts like activities, every aspect of the desktop seems to require tweaking in non obvious ways for it to be more usable, looks ok but not great.

XFCE

On Fedora, the default XFCE is very very basic, so much that I could hardly see a difference with one from 10 years ago. On Xubuntu, it's much much better. When I came to it from Gnome-Shell, I was surprised at how good was the "old" desktop paradigm for productivity. I also surprisingly found multiple desktops more natural to use than on Gnome Shell/KDE.
On Fedora the way to make it like Xubuntu is to install elementary icons, the whisker menu and choose the greybird/bluebird themes.

yum groups install "Xfce Desktop"
yum install xfce4-mixer.x86_64 xfce4-whiskermenu-plugin.x86_64 xfce4-cpugraph-plugin.x86_64 xfce4-mount-plugin.x86_64 xfce4-icon-theme.noarch google-droid* elementary-xfce-icon-theme.noarch xfce4-volumed.x86_64 pavucontrol.x86_64

Positive aspects: fast and lean, great start menu.
Negative aspects: external monitor support could be more automatic like Gnome-Shell, no nice overview of all windows, default installation can be a bit too bare, sometimes not sexy (volume applet is ugly, xubuntu provides the unity indicators in xfce as a remedy), primitive OSD.

Cinnamon, Unity, Conclusion

I gave a short try to cinnamon as well, in hopes that it was more stable than gnome shell. In short, it was not. It's certainly less of a memory hog, but I had some strange behavior with an additional phantom panel sometimes appearing at the bottom at the screen. And overall it looks a lot less polished.

Unity is more interesting, but it's too Ubuntu centric, I don't like the start button equivalent (slow, badly presented, don't care about HUD), the windows overview is not as useful as Gnome shell, the dock, something I usually like, is strangely annoying.

This is a very subjective review, my feeling is that in 2014, people should not waste their time with KDE or Cinnamon. Gnome shell could be worth a try if you don't care so much about memory leaks and slight instability but value a distraction free desktop. Otherwise go for XFCE or Unity on (X)ubuntu.

Two SABR for the same smile

Tue, 20 May 2014 12:08:00 +0000

While playing around with differential evolution to calibrate SABR, I noticed that sometimes, several set of parameters can lead to a very similar smile, usually the good one is for relatively low vol of vol and the bad one is for relatively high vol of vol. I first looked for errors in my implementation, but it’s a real phenomenon.

I used the normal implied volatility formula with beta=1, then converted it to lognormal (Black) volatility. While it might not be a great idea to rely on the normal formula with beta=1, I noticed the same phenomenon with the arbitrage free PDE density approach, especially for long maturities. Interestingly, I did not notice such behavior before with other stochastic volatility models like Heston or Schobel-Zhu: I suspect it has to do with the approximations rather than with the true behavior of SABR.

Heston vs SABR slice by slice fit

Thu, 15 May 2014 22:06:00 +0000

Some people use Heston to fit one slice of a volatility surface. In this case, some parameters are clearly redundant. Still, I was wondering how it fared against SABR, which is always used to fit a slice. And what about Schobel-Zhu?

Aggregated error in fit per slice on 10 surfaces

With Heston, the calibration is actually slightly better with kappa=0, that is, without mean reversion, because the global optimization is easier and the mean reversion is fully redundant. It’s still quite remarkable that 3 parameters result in a fit as good as 5 parameters.

Quadratic Spline with Knots at Mid-Points

Wed, 14 May 2014 14:12:00 +0000

Two months ago, I looked at arbitrage free interpolation using piecewise-constant density. This is equivalent to a piecewise quadratic polynomial in call prices where each piece is centered around each call strike.

I wondered at the time what a quadratic spline would look like on this problem, as it should be very close in theory, except that we can ensure that it is C1, a condition for a good looking implied volatility.

On Interviewing Candidates for a Job

Wed, 23 Apr 2014 17:18:00 +0000

I am going to write a little about my experience and conclusions so far around interviewing a candidate for a software developer position or for a quant position, but it should be quite general.

At first, I used to ask interview questions I liked when I was myself a candidate. On the technical side, it would stuff like:

which design patterns do you know?
what’s your opinion on design patterns?<
what’s a virtual method?
any interesting algorithm you like?
which libraries did you use?

For a quant type I’d ask more specific questions:

Non-linear Option Pricing

Fri, 18 Apr 2014 22:18:00 +0000

I am currently reading the book “Nonlinear Option Pricing” by J. Guyon and P. Henry-Labordère. It’s quite interesting even if the first third is quite theoretical. For example they describe how to solve some not well defined non-linear parabolic PDE by relying on the parabolic envelope. They also explain why most problems lead to parabolic PDEs in finance.

The rest is a bit more practical. I stumbled upon an good remark regarding Longstaff-Schwartz: the algorithm as Longstaff and Schwarz describe it does not necessary lead to a low-biased estimate as they use future information (the paths they regress on) in the Monte-Carlo estimate. It was actually a subject of discussion with colleagues, and I analyzed the numerical impact in a simple use case in http://papers.ssrn.com/abstract=2262259 In short: it’s actually more precise to include the path, even if the estimate is not purely low biased anymore, but the bias is really small in practice.

5 Minutes of Xtend

Tue, 08 Apr 2014 17:37:00 +0000

There is a relatively new JVM based language, Xtend. Their homepage says “JAVA 10, TODAY!”, so I thought I would give it a try, I was especially interested in operator overloading support, and the fact that it compiles to Java code, not Java byte code.

Unfortunately, after 5 minutes with it, and pasting some non Java code in an xtend file, Eclipse hangs forever, even on restart. After creating another workspace, just to trash the new workspace a similar way. This is quite incredible for a nearly 2 years old project, on eclipse.org.

Building a more accurate basis point volatility formula

Sat, 05 Apr 2014 15:42:00 +0000

P. Jaeckel has defied the limits of accuracy with his latest Black-Scholes volatility solver, managing to also improve performance compared to his earlier solver "By Implication". Out of a silly exercise, I decided to try my hand for a more accurate Normal (or basis point) volatility solver.

In reality, the problem is much simpler in the Bachelier/Normal model. A very basic analysis of Bachelier formula shows that the problem can be reduced to a single variable, as Choi et al explain in their paper. So the problem is not really one of solving, but one of approximating (the inverse of) a function.

The first step to build that function is to actually have a highly accurate slow solver as reference. This is quite easy, I just started with Choi formula and used Halley's method to refine. In reality, Halley's method is already a bit overkill on this problem: it works impressively well, 1 iteration is enough to have an insane level of accuracy, only noticeable when one works in high precision arithmetic (for example 50 digits). For double precision, Newton's method would actually be enough - I initially thought that my Halley's implementation did not work as it produced the exact same output as Newton in double precision. Li proposes the use of the SOR method, which for this exercise, behaves very much like Newton's method.

I then followed the logic from Choi et al, but working directly with in-the-money call options instead of straddles. Straddles sound neat at first (hides that we work in-the-money), but it's actually useless for the algorithm. Choi et al. ignore half of the straddle range when they use their eta transform in the paper. One other change is the mapping itself, I found a better mapping for the call options (but not that far of Choi initial idea). Finally, because I am lazy, I did not go to the pain of finding a good rational fraction approximation along with the square root problem they describe, I just tried a Chebyshev polynomial.

Unfortunately, a single Chebyshev polynomial does not work well: even with a very large (1000) degree it's not very precise, so much that I thought that my transform was garbage. I had noticed by mistake, that on another part (negative) of the interval, the Chebyshev polynomial worked actually very well to approximate something related to the volatility of another option. Suddendly came to me the idea of, like Johnson does in his Faddeeva package, using N Chebyshev polynomials on N small intervals. This is like the big heavy hammer for which everything looks like nails, but it's actually very fast to evaluate as the degree of each polynomial can then be low (7), plus a table lookup (could be coded as switch statements if one really cares about such details). The slowest part is actually the call to the log function.

The final bit is the use of a Taylor approximation for my -u/log(1-u) transform as it is not all that accurate in double precision when u is near 0. And that produces the following graph

It is interesting to note that "solving" the b.p. vol is 10x faster than solving the Black vol.

I wrote a small paper around all this where you'll find the details as well as Matlab code.

Fast and Accurate Implied Volatility Solver

Wed, 19 Mar 2014 18:10:00 +0000

The calibration of a stochastic volatility model or a volatility surface parameterization (like SVI) involves minimizing the model options volatilities against market options volatilities. Often, the model computes an option price, not an implied volatility. It is therefore useful to have a fast way to invert that option price to get back the implied volatility that corresponds to it. Furthermore during the calibration procedure, the model option price can vary widely: it is convenient to have a robust implied volatility solver.

Arbitrage Free Interpolation of Option Prices using Piecewise Constant Density

Mon, 17 Mar 2014 15:25:00 +0000

Tension splines can produce in some cases arbitrage free C2 interpolation of options, but unfortunately this is not guaranteed. It turns out that, on some not so nice looking data, where the discrete probability density is not monotone but only positive, all previously considered interpolation fail (spline in volatility or variance, tension spline in log prices, harmonic spline on prices).

K	vol	put	b-slope	b-convexity
300.0	0.682	0.090	0.00e+00	0.00e+00
310.0	0.654	0.136	4.60e-03	0.00e+00
320.0	0.621	0.192	5.60e-03	1.00e-03
330.0	0.594	0.288	9.60e-03	4.00e-03
340.0	0.560	0.404	1.16e-02	2.00e-03
350.0	0.520	0.530	1.26e-02	1.00e-03
360.0	0.484	0.736	2.06e-02	8.00e-03
370.0	0.467	1.232	4.96e-02	2.90e-02
380.0	0.442	1.898	6.66e-02	1.70e-02
390.0	0.427	3.104	1.21e-01	5.40e-02
400.0	0.412	4.930	1.83e-01	6.20e-02

Possibly the simplest arbitrage free interpolation is to postulate the density as piecewise constant, ideally centered around each strike (if not centered, there is no guarantee that it will be positive). If a spline is used for interpolation, this means a quadratic spline. Unfortunately, because it is not C2, it then still fails to be arbitrage free.

C2 Arbitrage Free Interpolation with Tension Splines

Tue, 11 Mar 2014 17:05:00 +0000

In a previous post, I have explored the arbitrage free wiggles in the volatility surface that P. Jaeckel found in his paper. I showed that interpolating in log prices instead of prices was enough to remove the wiggles, but then, it appears that the interpolation is not guaranteed to be arbitrage free, even though it often is. On another example from P. Jaeckel paper, that I reproduced inaccurately but well enough, it is not.

Bachelier and Black-Scholes Fits of the Volatility Surface, what about SABR?

Fri, 07 Mar 2014 15:31:00 +0000

I always wondered if Bachelier was really worse than Black-Scholes in practice. As an experiment I fit various implied volatility surfaces with Bachelier and Black-Scholes and look at the average error in implied volatility by slice.

In theory Bachelier is appealing because slightly simpler: log returns are a bit more challenging to think about than returns. And it also takes indirectly into account the fact that OTM calls are less likely than OTM puts because of default risk, if you assume absorbing probability at strike 0.

Bachelier

Black-Scholes

The error is in general smaller for Black-Scholes for short expiries, and higher for long expiries when compared to Bachelier. Interestingly, in theory, the difference of the models is more pronounced for longer expiries. One would have imagined that for very short expiries Bachelier would be equivalent to Black, it is not the case in term of fitting the smile.

Displaced diffusion (mixing both Bachelier and Black linearly) allows to gain x2 accuracy.

Displaced Diffusion

What about SABR? Let's look at lognormal SABR, usually used for equities.

Lognormal SABR (beta=1)

The fit is much better for long expiries, but not so great for a few outliers for long maturities, it can be actually worse than the simple displaced diffusion. A normal SABR might fix that.

Normal SABR (beta=0) using Hagan lognormal formula

If one relies on the standard lognormal formula, the beta=0 SABR behaves very badly.

Normal SABR using Hagan normal formula

This is fixed using the normal (bpvol) Hagan formula directly. The fit is then better overall for long maturities as expected from Black-Scholes vs Bachelier behavior.

If one look in log scale, the conclusion is not so obvious: beta=1 produces a better fit for a majority even for long expiries, but worse for a few (30% in my case) outliers.

What's clear however, is that one should never use the Black implied volatility Hagan formula with beta=0. This leaves a question on displaced SABR. Is displaced SABR is better suited than SABR with varying beta?

Arbitrage Free Wiggles

Mon, 03 Mar 2014 17:13:00 +0000

Peter Jaeckel, in a recent paper (pdf), shows that something that sounds like a reasonable arbitrage free interpolation can produce wiggles in the implied volatility slice.

The interpolation in question is using some convexity preserving spline on call and put option prices directly and in strike, assuming those input prices are arbitrage free. This is very similar to Kahale interpolation (pdf).

It seemed too crazy for me so I had to try out his example. And using a harmonic spline, it does produce arbitrage free wiggles.

Wiggles in the implied volatility

If we look at the probability density (the curvature), the Harmonic spline maintains a positive density, nearly piecewise flat in log scale, while Hyman, because it preserves only monotonicity, has some negative density which are cut out from the graph.

Probability Density

In reality, if we look at the interpolation of prices in log scale, one can see that splines won't behave as expected at first on small numbers: they will give a much higher weight to the high values, producing something like a piecewise linear interpolation.

In reality, what one really wants for such data is to just interpolate the log prices with a spline, not the prices. This is the curve named "Log" in the graphs, where a simple cubic spline is used on the log prices, and fed to exp after interpolation.

Now it sounds like a reasonable arbitrage free interpolation would be to interpolate the discrete density log linearly, in a similar spirit as Hagan-West yield curve interpolation (pdf).

In general, if you interpolate very small numbers with a spline, you probably are doing something wrong.

Adjoint Delta for Monte-Carlo

Tue, 25 Feb 2014 18:37:00 +0000

In an earlier post, I have been quickly exploring adjoint differentiation in the context of analytical Black-Scholes. Today, I tried to mix it in a simple Black-Scholes Monte-Carlo as described in L. Capriotti paper, and measured the performance to compute delta compared to a numerical single sided finite difference delta.

I was a bit surprised that even on a single underlying, without any real optimization, adjoint delta was faster by a factor of nearly 40%. I suspect this is mostly due to exp evaluations being /2.

SVI on top of SABR

Thu, 20 Feb 2014 18:36:00 +0000

Several papers show that the limit for large strikes of Heston is SVI.

Interestingly, I stumbled onto a surface where the Hagan SABR fit was perfect as well as the SVI fit, while the Heston fit was not.

Originally, I knew that, on this data, the SVI fit was perfect. Until today, I just never tried to fit a lognormal SABR on the same data. I did a small test with random values of the SABR parameters alpha, rho, nu, and found out that in deed, the SVI fit is always perfect on SABR.

Is this just because the Taylor expansion of SVI will match the Taylor expansion of SABR up to the fourth order? It seems that the wings are also not too far off in general so there could be more to it.

Gatheral actually devised an SVI formula that uses SABR like variables in a talk here.

Of course, the reverse is not true. SVI has more parameters and provides in general a better fit than SABR.

Smart Initial Guess for Schobel-Zhu

Wed, 19 Feb 2014 18:57:00 +0000

With a small time expansion, it is easy to derive a reasonable initial guess, without resorting to some global minimizer.

Like Forde did for Heston, one can find the 5 Schobel-Zhu parameters through 5 points at coordinates (0,0), (x0,t1), (-x0,t1), (x0,t2), (-x0,t2), where x0 is a chosen the log-moneyness, for example, 0.1 and t1, t2 relatively short expiries (for example, 0.1, 0.25).

We can truncate the small time expansion so that the polynomial in (x,t) is fully captured by those 5 points. In practice, I have noticed that using a more refined expansion with more terms resulted not only in more complex formulas to lookup the original stochastic volatility parameters, but also in an increased error, because of the redundancy of parameters in the polynomial expansion. My previous Schobel-Zhu expansion becomes just:

A Look at Small Time Expansions for Heston

Wed, 12 Feb 2014 13:13:00 +0000

Small time expansions for Heston can be useful during the calibration of the implied volatility surface, in order to find an initial guess for a local minimizer (for example, Levenberg-Marquardt). Even if they are not so accurate, they capture the dynamic of the model parameters, and that is often enough.

In 2011, Forde et al. proposed a second order small time expansion around the money, which I found to work well for calibration. More recently, Lorig et al. proposed different expansions up to order-3 around the money. I already looked at the later in my previous post, applying the idea to Schobel-Zhu.

A Small-Time Schobel-Zhu Expansion

Mon, 10 Feb 2014 18:30:00 +0000

The paper implied vol for any local stochastic vol model from Lorig et al. presents a very generic and simple formula to compute implied volatility expansions up to order-2 (there is actually an order-3 formula available in their Mathematica CDF file).

I tried it on the Schobel-Zhu stochastic volatility model. This model is an interesting alternative to Heston. I found that, in practice, the implied volatility surface fit was as good, while the simulation under the QE scheme is quite faster (and simpler) than Heston. Here is the result of applying their technique on Schobel-Zhu:

An SSD instead of a laptop

Tue, 04 Feb 2014 15:45:00 +0000

Last week, my work laptop died after spilling out some water on its keyboard inadvertently. Fortunately, its SSD was intact.

As the laptop SSD connector (SATA) and power follow the desktop computers standards, and as I use Linux, I just plugged the SSD to my home desktop and booted off the SSD. I had to change slightly the X configuration but otherwise everything worked. Linux is great for that

The same way, I just plugged the SSD to the desktop at work and it worked. Instead of carrying a laptop, I carry now my small and ultra-light SSD.

Brownian Bridge or Not with Heston Quadratic Exponential QMC

Fri, 24 Jan 2014 19:35:00 +0000

At first I did not make use of the Brownian Bridge technique in Heston QMC, because the variance process is not simulated like a Brownian Motion under the Quadratic Exponential algorithm from Andersen.

It is, however, perfectly possible to use the Brownian Bridge on the asset process. Does it make a difference? In my small test, it does not seem to make a difference. An additional question would be, is it better to take first N for the asset and next N for the variance or vice versa or intertwined? Intertwined would seem the most natural (this is what I used without Brownian Bridge, but for simplicity I did Brownian bridge on first N).

By contrast, Schobel-Zhu QE scheme can make full use of the Brownian Bridge technique, in the asset process as well as in the variance process. Here is a summary of the volatility process under the QE scheme from van Haastrecht:

Another nice property of Schobel-Zhu is that the QE simulation is as fast as Euler, and therefore 2.5x faster than the Heston QE.

I calibrated the model to the same surface, and the QMC price of a ATM call option seems to have a similar accuracy as Heston QMC. But we can see that the Brownian Bridge does increase accuracy in this case. I was surprised that accuracy was not much better than Heston, but maybe it is because I did yet not implement the Martingale correction, while I did so in the Heston case.

Adjoint Algorithmic Differentiation for Black-Scholes

Tue, 21 Jan 2014 13:03:00 +0000

Adjoint algorithmic differentiation is particularly interesting in finance as we often encounter the case of a function that takes many input (the market data) and returns one output (the price) and we would like to also compute sensitivities (greeks) to each input.

As I am just starting around it, to get a better grasp, I first tried to apply the idea to the analytic knock out barrier option formula, by hand, only to find out I was making way too many errors by hand to verify anything. So I tried the simpler vanilla Black-Scholes formula. I also made various errors, but managed to fix all of them relatively easily.

I decided to compare how much time it took to compute price, delta, vega, theta, rho, rho2 between single sided finite difference and the adjoint approach. Here are the results for 1 million options:

FD time=2.13s
Adjoint time=0.63s

It works well, but doing it by hand is crazy and too error prone. It might be simpler for Monte-Carlo payoffs however.

There are not many Java tools that can do reverse automatic differentiation, I found some thesis on it, with an interesting byte code oriented approach (one difficulty is that you need to reverse loops, while statements).

Placing the Strike on the Grid and Payoff Smoothing in Finite Difference Methods for Vanilla Options

Sun, 12 Jan 2014 16:27:00 +0000

Pooley et al., in Convergence Remedies for non-smooth payoffs in option pricing suggest that placing the strike on the grid for a Vanilla option is good enough:

At the same time, Tavella and Randall show in their book that numerically, placing the strike in the middle of two nodes leads to a more accurate result. My own numerical experiments confirm Tavella and Randall suggestion.

In reality, what Pooley et al. really mean, is that quadratic convergence is maintained if the strike is on the grid for vanilla payoffs, contrary to the case of discontinuous payoffs (like digital options) where the convergence decreases to order 1. So it's ok to place the strike on the grid for a vanilla payoff, but it's not optimal, it's still better to place it in the middle of two nodes. Here are absolute errors in a put option price:

on grid, no smoothing         0.04473021824995271
on grid, Simpson smoothing    0.003942854282069419
on grid, projection smoothing 0.044730218065351934
middle, no smoothing          0.004040359609906119

As expected (and mentioned in Pooley et al.), the projection does not do anything here. When the grid size is doubled, the convergence ratio of all methods is the same (order 2), but placing the strike in the middle still increases accuracy significantly.

Here is are the same results, but for a digital put option:
on grid, no smoothing 0.03781319921461046
on grid, Simpson smoothing    8.289052335705427E-4
on grid, projection smoothing 1.9698293587372406E-4
middle, no smoothing          3.5122153011418744E-4

Here only the 3 last methods are of order-2 convergence, and projection is in deed the most accurate method, but placing the strike in the middle is really not that much worse, and much simpler.

A disadvantage of Simpson smoothing (or smoothing by averaging), is that it breaks put-call parity (see the paper "Exact Forward and Put Call Parity with TR-BDF2")

I think the emphasis in their paper on "no smoothing is required" for vanilla payoffs can be misleading. I hope I have clarified it in this post.

Coordinate Transform of the Andreasen Huge SABR PDE & Spline Interpolation

Wed, 08 Jan 2014 18:51:00 +0000

Recently, I noticed how close are the two PDE based approaches from Andreasen-Huge and Hagan for an arbitrage free SABR. Hagan gives a local volatility very close to the one Andreasen-Huge use in the forward PDE in call prices. A multistep Andreasen-Huge (instead of their one step PDE method) gives back prices and densities nearly equal to Hagan density based approach.

Hagan proposed in some unpublished paper a coordinate transformation for two reasons: the ideal range of strikes for the PDE can be very large, and concentrating the points where it matters should improve stability and accuracy. The transform itself can be found in the Andersen-Piterbarg book “Interest Rate Modeling”, and is similar to the famous log transform, but for a general local volatility function (phi in the book notation).

Random Hardware Issues

Mon, 06 Jan 2014 18:44:00 +0000

Today, after wondering why my desktop computer became so unstable (frequent crashes under Fedora), I found out that the micro usb port of my cell phone has some kind of short circuit. My phone behaves strangely in 2014, it lasted nearly 1 week on battery (I lost it for half of the week), and seems to shutdown for no particular reason once in a while.

On the positive side, I also discovered, after owning my monitor for around 5 years, that it has SD card slots on the side, as well as USB ports. I always used the USB ports of my desktop and never really looked at the side of my monitor…

Levenberg Marquardt & Constraints by Domain Transformation

Tue, 17 Dec 2013 15:27:00 +0000

The Fortran minpack library has a good Levenberg-Marquardt minimizer, so good, that it has been ported to many programming languages. Unfortunately it does not support contraints, even simple bounds.

One way to achieve this is to transform the domain via a bijective function. For example, $a+\frac{b-a}{1+e^{-\alpha t}}$ will transform $]-\infty, +\infty[$ to ]a,b[. Then how should one choose $\alpha$?

A large $\alpha$ will make tiny changes in $t$ appear large. A simple rule is to ensure that $t$ does not create large changes in the original range ]a,b[, for example we can make $\alpha t \leq 1$, that is $ \alpha t= \frac{t-a}{b-a} $.

In practice, for example in the calibration of the Double-Heston model on real data, a naive $ \alpha=1 $ will converge to a RMSE of 0.79%, while our choice will converge to a RMSE of 0.50%. Both will however converge to the same solution if the initial guess is close enough to the real solution. Without any transform, the RMSE is also 0.50%. The difference in error might not seem large but this results in vastly different calibrated parameters. Introducing the transform can significantly change the calibration result, if not done carefully.

Another simpler way would be to just impose a cap/floor on the inputs, thus ensuring that nothing is evaluated where it does not make sense. In practice, it however will not always converge as well as the unconstrained problem: the gradient is not defined at the boundary. On the same data, the Schobel-Zhu, unconstrained converges with RMSE 1.08% while the transform converges to 1.22% and the cap/floor to 1.26%. The Schobel-Zhu example is more surprising since the initial guess, as well as the results are not so far:

Initial volatility (v0)	18.1961174789
Long run volatility (theta)	1
Speed of mean reversion (kappa)	101.2291161766
Vol of vol (sigma)	35.2221829015
Correlation (rho)	-73.7995231799
ERROR MEASURE	1.0614889526

Initial volatility (v0)	17.1295934569
Long run volatility (theta)	1
Speed of mean reversion (kappa)	67.9818356373
Vol of vol (sigma)	30.8491256097
Correlation (rho)	-74.614636128
ERROR MEASURE	1.2256421987

The initial guess is kappa=61% theta=11% sigma=26% v0=18% rho=-70%. Only the kappa is different in the two results, and the range on the kappa is (0,2000) (it is expressed in %), much larger than the result. In reality, theta is the issue (in (0,1000)). Forbidding a negative theta has an impact on how kappa is picked. The only way to be closer

Finally, a third way is to rely on a simple penalty: returning an arbitrary large number away from the boundary. In our examples this was no better than the transform or the cap/floor.

Trying out the various ways, it seemed that allowing meaningless parameters, as long as they work mathematically produced the best results with Levenberg-Marquardt, particularly, allowing for a negative theta in Schobel-Zhu made a difference.

Arbitrage Free SABR - Another View on Hagan Approach

Sat, 14 Dec 2013 00:56:00 +0000

Several months ago, I took a look at two interesting recent ways to price under SABR with no arbitrage:

One way is due to Andreasen and Huge, where they find an equivalent local volatility expansion, and then use a one-step finite difference technique to price.
The other way is due to Hagan himself, where he numerically solves an approximate PDE in the probability density, and then price with options by integrating on this density.

It turns out that the two ways are much closer than I first thought. Hagan PDE in the probability density is actually just the Fokker-Planck (forward) equation.

The $\alpha D(F)$ is just the equivalent local volatility. Andreasen and Huge use nearly the same local volatility formula but without the exponential part (that is often negligible except for long maturities), directly in Dupire forward PDE:

A common derivation (for example in Gatheral book of the Dupire forward PDE is to actually use the Fokker-Planck equation in the probability density integral formula. Out of curiosity, I tried to price direcly with Dupire forward PDE and the Hagan local volatility formula, using just linear boundary conditions. Here are the results on Hagan own example:

The Local Vol direct approach overlaps the Density approach nearly exactly, except at the high strike boundary, when it comes to probability density measure or to implied volatility smile. On Andreasen and Huge data, it gives the following:

American Option on Forward/Futures

Thu, 21 Nov 2013 11:17:00 +0000

Prices of a Future contract and a Forward contract are the same under the Black-Scholes assumptions (deterministic rates) but the price of options on Futures or options on Forwards might still differ. I did not find this obvious at first.

For example, when the underlying contract expiration date (Futures, Forward) is different from the option expiration date. For a Future Option, the Black-76 formula can be used, the discounting is done from the option expiry date, because one receives the cash on expiration due to the margin account. For a Forward Option, the discounting need to be done from the Forward contract expiry date.

Spikes in Heston/Schobel-Zhu Local Volatility

Wed, 20 Nov 2013 13:33:00 +0000

Using precise vanilla option pricing engine for Heston or Schobel-Zhu, like the Cos method with enough points and a large enough truncation can still lead to spikes in the Dupire local volatility (using the variance based formula).

Local volatility

Implied volatility

The large spikes in the local volatility 3d surface are due to constant extrapolation, but there are spikes even way before the extrapolation takes place at longer maturities. Even if the Cos method is precise, it seems to be not enough, especially for large strikes so that the second derivative over the strike combined with the first derivative over time can strongly oscillate.

Local Stochastic Volatility with Monte-Carlo

Wed, 16 Oct 2013 16:14:00 +0000

I always imagined local stochastic volatility to be complicated, and thought it would be very slow to calibrate.

After reading a bit about it, I noticed that the calibration phase could just consist in calibrating independently a Dupire local volatility model and a stochastic volatility model the usual way.

One can then choose to compute on the fly the local volatility component (not equal the Dupire one, but including the stochastic adjustment) in the Monte-Carlo simulation to price a product.

There are two relatively simple algorithms that are remarkably similar, one by Guyon and Henry-Labordère in "The Smile Calibration Problem Solved":

And one from Van der Stoep, Grzelak & Oosterlee "The Heston Stochastic-Local Volatility Model: Efficient Monte Carlo Simulation":

In the particle method, the delta function is a regularizing kernel approximating the Dirac function. If we use the notation of the second paper, we have a = psi.

The methods are extremely similar, the evaluation of the expectation is slightly different, but even that is not very different. The disadvantage is that all paths are needed at each time step. As a payoff is evaluated over one full path, this is quite memory intensive.

Heston, Schobel-Zhu, Bates, Double-Heston Fit

Mon, 07 Oct 2013 19:35:00 +0000

I did some experiments fitting Heston, Schobel-Zhu, Bates and Double-Heston to a real world equity index implied volatility surface. I used a global optimizer (differential evolution).

To my surprise, the Heston fit is quite good: the implied volatility error is less than 0.42% on average. Schobel-Zhu fit is also good (0.47% RMSE), but a bit worse than Heston. Bates improves quite a bit on Heston although it has 3 more parameters, we can see the fit is better for short maturities (0.33% RMSE). Double-Heston has the best fit overall but it is not that much better than simple Heston while it has twice the number of parameters, that is 10 (0.24 RMSE). Going beyond, for example Triple-Heston, does not improve anything, and the optimization becomes very challenging. With Double-Heston, I already noticed that kappa is very low (and theta high) for one of the processes, and kappa is very high (and theta low) for the other process: so much that I had to add a penalty to enforce constraints in my local optimizer. The best fit is at the boundary for kappa and theta. So double Heston already feels over-parameterized.

Heston volatility error

Schobel-Zhu volatility error

Bates volatility error

Double Heston volatility error

Another advantage of Heston is that one can find tricks to find a good initial guess for a local optimizer.

Update October 7: My initial fit relied only on differential evolution and was not the most stable as a result. Adding Levenberg-Marquardt at the end stabilizes the fit, and improves the fit a lot, especially for Bates and Double Heston. I updated the graphs and conclusions accordingly. Bates fit is not so bad at all.

Second Cumulant of Heston

Thu, 03 Oct 2013 17:27:00 +0000

I recently stumbled upon an error in the various papers related to the Heston Cos method regarding the second cumulant. It is used to define the boundaries of the Cos method. Letting phi be Heston characteristic function, the cumulant generating function is: $$g(u) = \log(\phi(-iu))$$

And the second cumulant is defined a: $$c_2 = g’’(0)$$

Compared to a numerical implementation, the c_2 from the paper is really off in many use cases.

Maxima for Symbolic Calculus

Wed, 02 Oct 2013 15:06:00 +0000

A few years ago, I found an interesting open source symbolic calculus software called Xcas. It can however be quickly limited, for example, it does not seem to work well to compute Taylor expansions with several embedded functions. Google pointed me to another popular open source package, Maxima. It looks a bit rudimentary (command like interface), but formulas can actually be very easily exported to latex with the tex command. Here is a simple example:

Martin Odersky teaches Scala to the Masses

Tue, 17 Sep 2013 20:11:00 +0000

I tried today the Scala courses on coursera by the Scala creator, Martin Odersky. I was quite impressed by the quality: I somehow believed Scala to be only a better Java, now I think otherwise. Throughout the course, even though it all sounds very basic, you understand the key concepts of Scala and why functional programming + OO concepts are a natural idea. What’s nice about Scala is that it avoids the functional vs OO or even the functional vs procedural debate by allowing both, because both can be important, at different scales. Small details can be (and probably should be) procedural for efficiency, because a processor is a processor, but higher level should probably be more functional (immutable) to be clearer, easier to evolve and more easily parallelized.

Setting Values in Java Enum - A Bad Idea

Thu, 12 Sep 2013 10:06:00 +0000

My Scala habits have made me create a stupid bug related to Java enums. In Scala, the concept of case classes is very neat and recently, I just confused enum in Java with what I sometimes do in Scala case classes.

I wrote an enum with a setter like:

    public static enum BlackVariateType {
        V0,
        ZERO_DERIVATIVE;

        private double volSquare;
        public double getBlackVolatilitySquare() {
            return volSquare;
        }

        public void setBlackVolatilitySquare(double volSquare) {
            this.volSquare = volSquare;
        }
   }

Here, calling setBlackVolatilitySquare will override any previous value, and thus, if several parts are calling it with different values, it will be a mess as there is only a single instance.

I am not sure if there is actually one good use case to have a setter on an enum. This sounds like a very dangerous practice in general. Member variables allowed should be only final.

Making Classic Heston Integration Faster than the Cos Method

Thu, 05 Sep 2013 17:35:00 +0000

A coworker pointed to me that Andersen and Piterbarg book “Interest Rate Modeling” had a chapter on Fourier integration applied to Heston. The authors rely on the Lewis formula to price vanilla call options under Heston.

Lewis formula

More importantly, they strongly advise the use of a Black-Scholes control variate. I had read about that idea before, and actually tried it in the Cos method, but it did not improve anything for the Cos method. So I was a bit sceptical. I decided to add the control variate to my Attari code. The results were very encouraging. So I pursued on implementing the Lewis formula and their basic integration scheme (no change of variable).

Attari formula

Carr-Madan formula (used by Lord-Kahl)

Heston formula

Cos formula

Better Fonts in Fedora than in Ubuntu

Sat, 31 Aug 2013 13:36:00 +0000

By default, text in Fedora 19 looks ok but not great. There are good tips to improve fonts on http://fedorasolved.org/Members/khaytsus/improve-fonts

As root user here is what need to be done:

cd /etc/fonts/conf.d

ln -s /usr/share/fontconfig/conf.avail/10-autohint.conf

ln -s /usr/share/fontconfig/conf.avail/11-lcdfilter-default.conf

yum install freetype-freeworld

Reboot, then, in gnome-tweak-tool choose hinting = slight, and antialiasing = rgba. I also choose the liberation sans and liberation mono fonts.

I am not sure which one did the trick, but it looks much better (even better than my laptop under Ubuntu, not sure why).

Attari, Lord-Kahl & Cos Methods Comparison on Heston

Wed, 28 Aug 2013 17:54:00 +0000

I recently wrote about the Cos method. While rereading the various papers on Heston semi-analytical pricing, especially the nice summary by Schmelzle, it struck me how close were the Attari/Bates methods and the Cos method derivations. I then started wondering if Attari was really much worse than the Cos method or not.

I noticed that Attari method accuracy is directly linked to the underlying Gaussian quadrature method accuracy. I found that the doubly adaptive Newton-Cotes quadrature by Espelid (coteda) was the most accurate/fastest on this problem (compared to Gauss-Laguerre/Legendre/Extrapolated Simpson/Lobatto). If the accuracy of the integration is 1e-6, Attari maximum accuracy will also be 1E-6, this means that very out of the money options will be completely mispriced (might even be negative). In a sense it is similar to what I observed on the Cos method.

Giving Fedora Another Chance

Wed, 14 Aug 2013 22:15:00 +0000

I have had some stability issues with the Ubuntu 13.04 on my home computer, not on my laptop. It might be related to hard disk encryption (out of curiosity I encrypted my main hard drive in the installer option, resulting in a usable but quite slow system - it’s incredible how much the hard drive is still important for performance). I did not have any particular issue on my work laptop with it.

Julia and the Cumulative Normal Distribution

Tue, 13 Aug 2013 15:52:00 +0000

I just stumbled upon Julia, a new programming language aimed at numerical computation. It’s quite new but it looks very interesting, with the promise of C like performance (thanks to LLVM compilation) with a much nicer syntax and parallelization features.Out of curiosity, I looked at their cumulative normal distribution implementation. I found that the (complimentary) error function (directly related to the cumulative normal distribution) algorithm relies on an algorithm that can be found in the Faddeeva library. I had not heard of this algorithm or this library before, but the author, Steven G. Johnson, claims it is faster and as precise as Cody & SLATEC implementations. As I previously had a look at those algorithms and was quite impressed by Cody’s implementation.The source of Faddeeva shows a big list (100) of Chebychev expansions for various ranges of a normalized error function. I slightly modified the Faddeva code to compute directly the cumulative normal distribution, avoiding some exp(-xx)exp(xx) calls on the way.Is it as accurate? I compared against a high precision implementation as in my previous test of cumulative normal distribution algorithms. And after replacing the exp(-xx) with Cody’s trick to compute it with higher accuracy, here is how it looks (referenced as “Johnson”).I also measured performance on various ranges, and found out that this Johnson algorithm is around 2x faster than Cody (in Scala) and 30% faster than my optimization of Cody (using a table of exponentials for Cody’s trick).

The COS method for Heston

Fri, 02 Aug 2013 14:19:00 +0000

Fang, in her thesis, has the idea of the COS method and applies it to Heston. There are several published papers around it to price options under various models that have a known characteristic function, as well as to price more exotic options like barriers or bermudans.

The COS method is very close to the more standard Heston quasi analytic formula (use transform of characteristic function for the density and integrates the payoff with the density, exchanging summation), except that the more simple Fourier series are used instead of the standard Fourier transform. As a consequence there are a few more approximations that are done related to the truncation of the domain of integration and the result is already discrete, so no need for a Gaussian quadrature.

Octave vs Scilab for PDEs in Finance

Tue, 30 Jul 2013 12:10:00 +0000

I was used to Scilab for small experiments involving linear algebra. I also like some of Scilab choices in algorithms: for example it provides PCHIM monotonic spline algorithm, and uses Cody for the cumulative normal distribution.

Matlab like software is particularly well suited to express PDE solvers in a relatively concise manner. To illustrate some of my experiments, I started to write a Scilab script for the Arbitrage Free SABR problem. It worked nicely and is a bit nicer to read than my equivalent Scala program. But I was a bit surprised by the low performance.

Joda LocalDateTime vs DateTime

Wed, 17 Jul 2013 14:11:00 +0000

Joda has the concept of LocalDate LocalDateTime and DateTime. The LocalDate is just a simple date, while DateTime is a date and a time zone.

Where I work we have a similar distinction, although not the same: a simple “absolute” date object without time vs a relative date (a timestamp) like the JDK Date.

The standard JDK Date class is a date without a time zone, but Sun deprecated in JDK 1.1 all methods allowing to use it like a LocalDate, forcing to use it through a Calendar (i.e. like a DateTime), that is, with a TimeZone.

The CUDA Performance Myth II

Fri, 12 Jul 2013 15:23:00 +0000

This is a kind of following to the CUDA performance myth. There is a recent news on the java concurrent mailing list about SplittableRandom class proposed for JDK8. It is a new parallel random number generator a priori usable for Monte-Carlo simulations.

It seems to rely on some very recent algorithm. There are some a bit older ones: the ancestor, L’Ecuyer MRG32k3a that can be parallelized through relatively costless skipTo methods, a Mersenne Twister variant MTGP, and even the less rigourous XorWow popularized by NVidia CUDA.

Unity vs KDE in Virtualbox

Wed, 10 Jul 2013 23:17:00 +0000

The other day I installed the latest Ubuntu 13.04 under a VirtualBox virtual machine using Windows as host. To my surprise, unity failed to launch properly on the virtual machine reboot, with compiz complaining, something I have sometimes seen on my work laptop. It’s more surprising in a VM since it is in a way much more standard (no strange graphic card, no strange driver, the same stuff for every VirtualBox user (maybe I’m wrong there?)). I therefore installed KDE as a way to bypass this issue. Not only it worked, but the UI was much faster: there was some very noticeable lag in Unity, slow fade in fade out effects, when it worked before the reboot.

Bessel and Harmonic Kinks in the Forward

Tue, 02 Jul 2013 15:44:00 +0000

As Bessel (sometimes called Hermite) spline interpolation is only C1, like the Harmonic spline from Fritsch-Butland, the forward presents small kinks compared to a standard cubic spline. Hyman filtering also creates a kink where it fixes the monotonicity. Those are especially visible with a log scale in time. Here is how it looks on the Hagan-West difficult curve.

Godaddy sold my domain name

Mon, 24 Jun 2013 19:11:00 +0000

I discovered that suddenly emails sent to me bounced back yesterday. I logged in my godaddy account and to my surprise saw that I did not own any domain name anymore. I looked at my emails to see if I had received a warning as is usually the case when your domain is about to expire. There was none recent, the most recent was from may 2011, the last time I had renewed my domain.

Scala Build Tool : SBT

Wed, 19 Jun 2013 18:01:00 +0000

It’s been a while since I do a pet project in Scala, and today, after many trials before, I decided to give another go at Jetbrain Idea for Scala development, as Eclipse with the Scala plugin tended to crash a little bit too often for my taste (resulting sometimes in loss of a few lines of code). I could have just probably updated eclipse and the scala plugin, mine were not very old, but not the latest.

The Finite Difference Theta Scheme Optimal Theta

Tue, 18 Jun 2013 15:02:00 +0000

The theta finite difference scheme is a common generalization of Crank-Nicolson. In finance, the book from Wilmott, a paper from A. Sepp, one from Andersen-Ratcliffe present it. Most of the time, it’s just a convenient way to handle implicit $\theta=1$, explicit $\theta=0$ and Crank-Nicolson $\theta=0.5$ with the same algorithm.

Wilmott makes an interesting remark: one can choose a theta that will cancel out higher order terms in the local truncation error and therefore should lead to increased accuracy. $$\theta = \frac{1}{2}- \frac{(\Delta x)^2}{12 b \Delta t} $$ where $b$ is the diffusion coefficient. This leads to $\theta < \frac{1}{2}$, which means the scheme is not unconditionally stable anymore but needs to obey (see Morton & Mayers p 30): $$b \frac{\Delta t}{(\Delta x)^2} \leq \frac{5}{6}$$ and to ensure that $\theta \geq 0 $: $$b \frac{\Delta t}{(\Delta x)^2} \geq \frac{1}{6}$$

Simple 'Can Scala Do This?' Questions

Tue, 11 Jun 2013 00:28:00 +0000

Today, a friend asked me if Scala could pass primitives (such as Double) by reference. It can be useful sometimes instead of creating a full blown object. In Java there is commons lang MutableDouble. It could be interesting if there was some optimized way to do that.

One answer could be: it’s not functional programming oriented and therefore not too surprising this is not encouraged in Scala.

Then he wondered if we could use it for C#.

Akima for Yield Curve Interpolation ?

Mon, 03 Jun 2013 00:07:00 +0000

On my test of yield curve interpolations, focusing on parallel delta versus sequential delta, Akima is the worst of the lot. I am not sure why this interpolation is still popular when most alternatives seem much better. Hyman presented some of the issues with Akima in his paper in 1983.

In the following graph, a higher value is a higher parallel-vs-sequential difference.

That plus the Hagan-West example of a tricky curve looks a bit convoluted with it (although it does not have any negative forward).

I have used Quantlib implementation, those results make me wonder if there is not something wrong with the boundaries.

2 Ways for an Accurate Barrier with Finite Difference

Sun, 02 Jun 2013 00:46:00 +0000

I had explored the issue of pricing a barrier using finite difference discretization of the Black-Scholes PDE a few years ago. Briefly, for explicit schemes, one just need to place the barrier on the grid and not worry about much else, but for implicit schemes, either the barrier should be placed on the grid and the grid truncated at the barrier, or a fictitious point should be introduced to force the correct price at the barrier level (0, typically).

The fictitious point approach is interesting for the case of varying rebates, or when the barrier moves around. I first saw this idea in the book "Paul Wilmott on Quantitative Finance".

Recently, I noticed that Hagan made use of the ficitious point approach in its "Arbitrage free SABR" paper, specifically he places the barrier in the middle of 2 grid points. There is very little difference between truncating the grid and the fictitious point for a constant barrier.

In this specific case there is a difference because there are 2 additional ODE solved on the same grid, at the boundaries. I was especially curious if one could place the barrier exactly at 0 with the fictitious point, because then one would potentially need to evaluate coefficients for negative values. It turns out you can, as values at the fictitious point are actually not used: the mirror point inside is used because of the mirror boundary conditions.

So the only difference is the evaluation of the first derivative at the barrier (used only for the ODE): the fictitious point uses the value at barrier+h/2 where h is the space between two points at the same timestep, while the truncated barrier uses a value at barrier+h (which can be seen as standard forward/backward first order finite difference discretization at the boundaries). For this specific case, the fictitious point will be a little bit more precise for the ODE.

SABR with the new Hagan PDE Approach

Tue, 28 May 2013 15:56:00 +0000

At a presentation of the Thalesians, Hagan has presented a new PDE based approach to compute arbitrage free prices under SABR. This is similar in spirit as Andreasen-Huge, but the PDE is directly on the density, not on the prices, and there is no one-step procedure: it's just like a regular PDE with proper boundary conditions.

I was wondering how it compared to Andreasen Huge results.

My first implementation was quite slow. I postulated it was likely the Math.pow function calls. It turns out they could be reduced a great deal. As a result, it's now quite fast. But it would still be much slower than Andreasen Huge. Typically, one might use 40 time steps, while Andreasen Huge is 1, so it could be around a 40 to 1 ratio. In practice it's likely to be less than 10x slower, but still.

While looking at the implied volatilities I found something intriguing with Andreasen Huge: the implied volatilities from the refined solution using the corrected forward volatility look further away from the Hagan implied volatilitilies than without adjustment, and it's quite pronounced at the money.

Interestingly, the authors don't plot that graph in their paper. They plot a similar graph of their own closed form analytic formula, that is in reality used to compute the forward volatility. I suppose that because they calibrate and price through their method, they don't really care so much that the ATM prices don't match Hagan original formula.

We can see something else on that graph: Hagan PDE boundary is not as nice as Andreasen Huge boundary for high strikes (they use a Hagan like approx at the boundaries, this is why it crosses the Hagan implied volatilities there).

If we use a simple option gamma = 0 boundary in Andreasen Huge, this results in a very similar shape as the Hagan PDE. This is because the option price is effectively 0 at the boundary.
Hagan chose a specifically taylored Crank-Nicolson scheme. I was wondering how it fared when I reduced the number of time-steps.

The answer is: not good. This is the typical Crank-Nicolson issue. It could be interesting to adapt the method to use Lawson-Morris-Goubet or TR-BDF2, or a simple Euler Richardson extrapolation. This would allow to use less time steps, as in practice, the accuracy is not so bad with 10 time steps only.

What I like about the Hagan PDE approach is that the implied vols and the probability density converge well to the standard Hagan formula, when there is no negative density problem, for example for shorter maturities. This is better than Andreasen Huge, where there seems to be always 1 vol point difference. However their method is quite slow compared to the original simple analytic formula.

Update March 2014 - I have now a paper around this "Finite Difference Techniques for Arbitrage Free SABR"

SABR with Andreasen-Huge

Fri, 24 May 2013 14:17:00 +0000

I am on holiday today. Unfortunately I am still thinking about work-related matters, and out of curiosity, wanted to do a little experiment. I know it is not very good to spend free time on work related stuff: there is no reward for it, and there is so much more to life. Hopefully it will be over after this post.

Around 2 years ago, I saw a presentation from Andreasen and Huge about how they were able to price/calibrate SABR by a one-step finite difference technique. At that time, I did not understand much their idea. My mind was too focused on more classical finite differences techniques and not enough on the big picture in their idea. Their idea is quite general and can be applied to much more than SABR.

Recently there has been some talk and development going on where I work about SABR (a popular way to interpolate the option implied volatility surface for interest rate derivatives), especially regarding the implied volatility wings at low strike, and sometimes on how to price in a negative rates environment. There are actually quite a bit of research papers around this. I am not really working on that part so I just mostly listened. Then a former coworker suggested that the Andreasen Huge method was actually what banks seemed to choose in practice. A few weeks later, the Thalesians (a group for people interested in quantitative finance) announced a presentation by Hagan (one of the inventor of SABR) about a technique that sounded very much like Andreasen-Huge to deal with the initial SABR issues in low rates.

As the people working on this did not investigate Andreasen-Huge technique, I somehow felt that I had to and that maybe, this time, I would be able to grasp their idea.

It took me just a few hours to have meaningful results. Here is the price of out of the money vanilla options using alpha = 0.0758194, nu = 0.1, beta = 0.5, rho = -0.1, forward = 0.02, and a maturity of 2 years.

I did not have in my home library a way to find the implied volatility for a given price. I knew of 2 existing methods, Jaeckel "By Implication", and Li rational functions approach. I discovered that Li wrote a new paper on the subject where he uses a SOR method to find the implied volatility and claims it's very accurate, very fast and very robust. Furthermore, the same idea can be applied to normal implied volatility. What attracted me to it is the simplicity of the underlying algorithm. Jaeckel's way is a nice way to do Newton-Raphson, but there seems to be so many things to "prepare" to make it work in most cases, that I felt it would be too much work for my experiment. It took me a few more hours to code Li SOR solvers, but it worked amazingly well for my experiment.

At first I had an error in my boundary condition and had no so good results especially with a long maturity. The traps with Andreasen-Huge technique are very much the same as with classical finite differences: be careful to place the strike on the grid (eventually smooth it), and have good boundaries.

Large Steps in Schobel-Zhu/Heston the Lazy Way

Fri, 17 May 2013 12:46:00 +0000

Van Haastrecht, Lord and Pelsser present an effective way to price derivatives by Monte-Carlo under the Schobel-Zhu model (as well as under the Schobel-Zhu-Hull-White model). It's quite similar to Andersen QE scheme for Heston in spirit.

In their paper they evolve the (log) asset process together with the volatility process, using the same discretization times. A while ago, when looking at Joshi and Chan large steps for Heston, I noticed that, inspired by Broadie-Kaya exact Heston scheme, they present the idea to evolve the variance process using small steps and the asset process using large steps (depending on the payoff) using the integrated variance value computed by small steps. The asset steps correspond to payoff evaluation dates At that time I had applied this idea to Andersen QE scheme and it worked reasonably well.

So I tried to apply the same logic to Schobel Zhu, and my first tests show that it works too. Interestingly, the speed gain is about 2x. Here are the results for a vanilla call option of different strikes.

Similar Error between long and short asset steps

Long steps take around 1/2 the time to compute

I would have expected the difference in performance to increase when the step size is decreasing, but it's not the case on my computer.

It's not truly large steps like Joshi and Chan do in their integrated double gamma scheme as the variance is still discretized in relatively small steps in my case, but it seems like a good, relatively simple optimization. A while ago, I did also implement the full Joshi and Chan scheme, but it's really interesting if one is always looking for long steps: it is horribly slow when the step size is small, which might occur for many exotic payoffs, while Andersen QE scheme perform almost as well as log-Euler in terms of computational cost.

Exact Forward in Monte-Carlo

Mon, 13 May 2013 17:58:00 +0000

Where I work, there used to be quite a bit of a confusion on which rates one should use as input to a Local Volatility Monte-Carlo simulation.

In particular there is a paper in the Journal of Computation Finance by Andersen and Ratcliffe “The Equity Option Volatility Smile: a Finite Difference Approach” which explains one should use specially tailored rates for the finite difference scheme in order to reproduce exact Bond price and exact Forward contract prices

Quasi Monte Carlo in Finance

Mon, 13 May 2013 13:16:00 +0000

I have been wondering if there was any better alternative than the standard Sobol (+ Brownian Bridge) quasi random sequence generator for the Monte Carlo simulations of finance derivatives.

Here is what I found:

Scrambled Sobol. The idea is to rerandomize the quasi random numbers slightly. It can provide better uniformity properties and allows for a real estimate of the standard error. There are many ways to do that. The simple Cranley Patterson rotation consisting in adding a pseudo random number modulo 1, Owen scrambling (permutations of the digits) and simplifications of it to achieve a reasonable speed. This is all very well described in Owen Quasi Monte Carlo document
Lattice rules. It is another form of quasi random sequences, which so far was not very well adapted to finance problems. A presentation from Giles & Kuo look like it's changing.
Fast PCA. An alternative to Brownian Bridge is the standard PCA. The problem with PCA is the performance in O(n^2). A possible speedup is possible in the case of a equidistant time steps. This paper shows it can be generalized. But the data in it shows it is only advantageous for more than 1024 steps - not so interesting in Finance.

Time Estimates in Software Development

Tue, 07 May 2013 21:01:00 +0000

Recently, that I completed a project that I had initially estimated to around 2 months, in nearly 4 hours. This morning I fixed the few remaining bugs. I looked at the clock, surprised it was still so early and I still had so many hours left in the day.

Now I have more time to polish the details and go beyond the initial goal (I think this scares my manager a bit), but I could (and I believe some people do this often) stop now and all the management would be satisfied.

Upper Bounds in American Monte-Carlo

Tue, 30 Apr 2013 17:05:00 +0000

Glasserman and Yu (GY) give a relatively simple algorithm to compute lower and upper bounds of a the price of a Bermudan Option through Monte-Carlo.

I always thought it was very computer intensive to produce an upper bound, and that the standard Longstaff Schwartz algorithm was quite precise already. GY algorithm is not much slower than the Longstaff-Schwartz algorithm, but what's a bit tricky is the choice of basis functions: they have to be Martingales. This is the detail I overlooked at first and I, then, could not understand why my results were so bad. I looked for a coding mistake for several hours before I figured out that my basis functions were not Martingales. Still it is possible to find good Martingales for the simple Bermudan Put option case and GY actually propose some in another paper.

Here are some preliminary results where I compare the random number generator influence and the different methods. I include results for GY using In-the-money paths only for the regression (-In suffix) or all (no suffix).


value using 16k training path and Sobol - GY-Low-In is very close to LS.

error using 16k training path - a high number of simulations not that useful

error using 1m simulation paths - GY basis functions require less training than LS

One can see the the upper bound is not that precise compared to the lower bound estimate, and that using only in the money paths makes a big difference. GY regression is good with only 1k paths, LS requires 10x more.

Surprisingly, I noticed that the Brownian bridge variance reduction applied to Sobol was increasing the GY low estimate, so as to make it sometimes slightly higher than Longstaff-Schwartz price.

The Wonderful UN

Wed, 24 Apr 2013 21:35:00 +0000

Already the name United Nations should be suspicious, but now they are shown to have spread Cholera to Haiti, as if the country did not have enough suffering. They have a nice building in New-York, and used to have a popular representative, but unfortunately, for poor countries, they never really achieved much. In Haiti, there were many stories of rapes and corruption by U.N. members more than 10 years ago. The movie The Whistleblower suggests it was the same in the Balkans. I am sure it did not change much since.

Moved From Dropbox to Ubuntu One

Tue, 23 Apr 2013 20:44:00 +0000

Dropbox worked well, but the company decided to blacklist it. I suppose some people abused it. While looking for an alternative, I found Ubuntu One. It’s funny I never tried it before even though I use Ubuntu. I did not think it was a dropbox replacement, but it is. And you get 5GB instead of Dropbox 2GB limit, which is enough for me (I was a bit above the 2GB limit). It works well under Linux but as well on Android and there is an iOS app I have not yet tried. It also works on Windows and Mac.

Quasi Monte-Carlo & Longstaff-Schwartz American Option price

Mon, 22 Apr 2013 18:00:00 +0000

In the book Monte Carlo Methods in Financial Engineering, Glasserman explains that if one reuses the paths used in the optimization procedure for the parameters of the exercise boundary (in this case the result of the regression in Longstaff-Schwartz method) to compute the Monte-Carlo mean value, we will introduce a bias: the estimate will be biased high because it will include knowledge about future paths.

However Longstaff and Schwartz seem to just reuse the paths in their paper, and Glasserman himself, when presenting Longstaff-Schwartz method later in the book just use the same paths for the regression and to compute the Monte-Carlo mean value.

How large is this bias? What is the correct methodology?

I have tried with Sobol quasi random numbers to evaluate that bias on a simple Bermudan put option of maturity 180 days, exercisable at 30 days, 60 days, 120 days and 180 days using a Black Scholes volatility of 20% and a dividend yield of 6%. As a reference I use a finite difference solver based on TR-BDF2.

I found it particularly difficult to evaluate it: should we use the same number of paths for the 2 methods or should we use the same number of paths for the monte carlo mean computation only? Should we use the same number of paths for regression and for monte carlo mean computation or should the monte carlo mean computation use much more paths?

I have tried those combinations and was able to clearly see the bias only in one case: a large number of paths for the Monte-Carlo mean computation compared to the number of paths used for the regression using a fixed total number of paths of 256*1024+1, and 32*1024+1 paths for the regression.

FDM price=2.83858387194312
Longstaff discarded paths price=2.8385854695510426
Longstaff reused paths price=2.8386108892756847

Those numbers are too good to be a real. If one reduces too much the total number of paths or the number of paths for the regression, the result is not precise enough to see the bias. For example, using 4K paths for the regression leads to 2.83770 vs 2.83767. Using 4K paths for regression and only 16K paths in total leads to 2.8383 vs 2.8387. Using 32K paths for regressions and increasing to 1M paths in total leads to 2.838539 vs 2.838546.

For this example the Longstaff-Schwartz price is biased low, the slight increase due to path reuse is not very visible and most of the time does not deteriorate the overall accuracy. But as a result of reusing the paths, the Longstaff-Schwartz price might be higher than the real value.

A Fast Exponential Function in Java

Fri, 19 Apr 2013 16:48:00 +0000

In finance, because one often dicretize the log process instead of the direct process for Monte-Carlo simulation, the Math.exp function can be called a lot (millions of times for a simulation) and can be a bottleneck. I have noticed that the simpler Euler discretization was for local volatility Monte-Carlo around 30% faster, because it avoids the use of Math.exp.

Can we improve the speed of exp over the JDK one? At first it would seem that the JDK would just call either the processor exp using an intrinsic function call and that should be difficult to beat. However what if one is ok for a bit lower accuracy? Could a simple Chebyshev polynomial expansion be faster?

Out of curiosity, I tried a Chebyshev polynomial expansion with 10 coefficients stored in a final double array. I computed the coefficient using a precise quadrature (Newton-Cotes) and end up with 1E-9, 1E-10 absolute and relative accuracy on [-1,1].

Here are the results of a simple sum of 10M random numbers:

0.75s for Math.exp sum=1.7182816693332244E7
0.48s for ChebyshevExp sum=1.718281669341388E7
0.40s for FastMath.exp sum=1.7182816693332244E7

So while this simple implementation is actually faster than Math.exp (but only works within [-1,1]), FastMath from Apache commons maths, that relies on a table lookup algorithm is just faster (in addition to being more precise and not limited to [-1,1]).

Of course if I use only 5 coefficients, the speed is better, but the relative error becomes around 1e-4 which is unlikely to be satisfying for a finance application.

0.78s for Math.exp sum=1.7182816693332244E7
0.27s for ChebyshevExp sum=1.718193001875838E7
0.40s for FastMath.exp sum=1.7182816693332244E7

Root finding in Lord Kahl Method to Compute Heston Call Price (Part III)

Fri, 12 Apr 2013 13:41:00 +0000

I forgot two important points in my previous post about Lord-Kahl method to compute the Heston call price:

Scaling: scaling the call price appropriately allows to increase the maximum precision significantly, because the Carr-Madan formula operates on log(Forward) and log(Strike) directly, but not the ratio, and alpha is multiplied by the log(Forward). I simply scale by the spot, the call price is (S_0*max(S/S_0-K/S0)). Here are the results for Lord-Kahl, Kahl-Jaeckel (the more usual way limited to machine epsilon accuracy), Forde-Jacquier-Lee ATM implied volatility without scaling for a maturity of 1 day:

Strike	Lord-Kahl	Kahl-Jaeckel	Forde-Jacquier-Lee
62.5	2.919316809400033E-34	8.405720564041985E-12	0.0
68.75	-8.923683388191852E-28	1.000266536266281E-11	0.0
75.0	-3.2319611910032E-22	2.454925152051146E-12	0.0
81.25	1.9401743410877718E-16	2.104982854689297E-12	0.0
87.5	-Infinity	-1.6480150577535824E-11	0.0
93.75	Infinity	1.8277663826893331E-9	1.948392142070432E-9
100.0	0.4174318393886519	0.41743183938679845	0.4174314959743768
106.25	1.326968012594355E-11	7.575717830832218E-11	1.1186618909114702E-11
112.5	-5.205783145942609E-21	2.5307755890935368E-11	6.719872683111381E-45
118.75	4.537094156599318E-25	1.8911094912255066E-11	3.615356241778357E-114
125.0	1.006555799739525E-27	3.2365221613872563E-12	2.3126009701775733E-240
131.25	4.4339539263484925E-31	2.4794388764348696E-11	0.0

One can see negative prices and meaningless prices outside ATM. With scaling it changes to:

Root finding in Lord Kahl Method to Compute Heston Call Price (Part II)

Thu, 11 Apr 2013 16:29:00 +0000

In my previous post, I explored the Lord-Kahl method to compute the call option prices under the Heston model. One of the advantages of this method is to go beyond machine epsilon accuracy and be able to compute very far out of the money prices or very short maturities. The standard methods to compute the Heston price are based on a sum/difference where both sides are far from 0 and will therefore be limited to less than machine epsilon accuracy even if the integration is very precise.

Root finding in Lord Kahl Method to Compute Heston Call Price

Tue, 09 Apr 2013 19:49:00 +0000

I just tried to implement Lord Kahl algorithm to compute the Heston call price. The big difficulty of their method is to find the optimal alpha. That’s what make it work or break. The tricky part is that the function of alpha we want to minimize has multiple discontinuities (it’s periodic in some ways). This is why the authors rely on the computation of an alpha_max: bracketing is very important, otherwise your optimizer will jump the discontinuity without even noticing it, while you really want to stay in the region before the first discontinuity.

From Double Precision Normal Density to Double Precision Cumulative Normal Distribution

Tue, 02 Apr 2013 14:24:00 +0000

Marsaglia in his paper on Normal Distribution made the same mistake I initially did while trying to verify the accuracy of the normal density.In his table of values comparing the true value computed by Maple for some values of x to the values computed by Sun or Ooura erfc, he actually does not really use the same input for the comparison. One example is the last number: 16.6. 16.6 does not have an exact representation in double precision, even though it is displayed as 16.6 because of the truncation at machine epsilon precision. Using Python mpmath, one can see that:>>> mpf(-16.6)mpf(’-16.6000000000000014210854715202004’)This is the more accurate representation if one goes beyond double precision (here 30 digits). And the value of the cumulative normal distribution is:>>> ncdf(-16.6)mpf(‘3.4845465199503256054808152068743e-62’)It is different from:>>> ncdf(mpf("-16.6"))mpf(‘3.48454651995040810217553910503186e-62’)where in this case it is really evaluated around -16.6 (up to 30 digits precision). Marsaglia gives this second number as reference. But all the other algorithms will actually take as input the first input. It is more meaningful to compare results using the exact same input. Using human readable but computer truncated numbers is not the best. The cumulative normal distribution will often be computed using some output of some calculation where one does not have an exact human readable input.The standard code for Ooura and Schonfelder (as well as Marsaglia) algorithms for the cumulative normal distribution don’t use Cody’s trick to evaluate the exp(-xx). This function appears in all those implementations because it is part of the dominant term in the usual expansions. Out of curiosity, I replaced this part with Cody trick. For Ooura I also made minor changes to make it work directly on the CND instead of going through the error function erfc indirection. Here are the results without the Cody trick (except for Cody):and with it:All 3 algorithms are now of similiar accuracy (note the difference of scale compared to the previous graph), with Schonfelder being a bit worse, especially for x >= -20. If one uses only easily representable numbers (for example -37, -36,75, -36,5, …) in double precision then, of course, Cody trick importance won’t be visible and here is how the 3 algorithms would fare with or without Cody trick:Schonfelder looks now worse than it actually is compared to Cody and Ooura.To conclude, if someone claims that a cumulative normal distribution is up to double precision accuracy and it does not use any tricks to compute exp(-xx), then beware, it probably is quite a bit less than double precision.

Cracking the Double Precision Gaussian Puzzle

Fri, 22 Mar 2013 12:20:00 +0000

In my previous post, I stated that some library (SPECFUN by W.D. Cody) computes $e^{-\frac{x^2}{2}}$ the following way:

xsq = fint(x * 1.6) / 1.6;
del = (x - xsq) * (x + xsq);
result = exp(-xsq * xsq * 0.5) * exp(-del *0.5);

where fint(z) computes the floor of z.

Why 1.6?

An integer divided by 1.6 will be an exact representation of the corresponding number in double: 1.6 because of 16 (dividing by 1.6 is equivalent to multiplying by 10 and dividing by 16 which is an exact operation). It also allows to have something very close to a rounding function: x=2.6 will make xsq=2.5, x=2.4 will make xsq=1.875, x=2.5 will make xsq=2.5. The maximum difference between x and xsq will be 0.625.

A Double Precision Puzzle with the Gaussian

Wed, 20 Mar 2013 17:50:00 +0000

Some library computes the Gaussian density function $$e^{-\frac{x^2}{2}}$$ the following way:

xsq = fint(x * 1.6) / 1.6;
del = (x - xsq) * (x + xsq);
result = exp(-xsq * xsq * 0.5) * exp(-del *0.5);

where fint(z) computes the floor of z.

Basically, x*x is rewritten as xsq*xsq+del. I have seen that trick once before, but I just can’t figure out where and why (except that it is probably related to high accuracy issues).

The answer is in the next post.

A Seasoned Volatility Swap

Thu, 14 Mar 2013 19:55:00 +0000

This is very much what's in the Carr-Lee paper "Robust Replication of Volatility Derivatives", but it wasn't so easy to obtain in practice:

The formulas as written in the paper are not usable as is: they can be simplified (not too difficult, but intimidating at first)
The numerical integration is not trivial: a simple Gauss-Laguerre is not precise enough (maybe if I had an implementation with more points), a Gauss-Kronrod is not either (maybe if we split it in different regions). Funnily a simple adaptive Simpson works ok (but my boundaries are very basic: 1e-5 to 1e5).

A Volatility Swap and a Straddle

Tue, 12 Mar 2013 21:36:00 +0000

A Volatility swap is a forward contract on future realized volatility. The pricing of such a contract used to be particularly challenging, often either using an unprecise popular expansion in the variance, or a model specific way (like Heston or local volatility with Jumps). Carr and Lee have recently proposed a way to price those contracts in a model independent way in their paper “robust replication of volatility derivatives”. Here is the difference between the value of a synthetic volatility swap payoff at maturity (a newly issued one, with no accumulated variance) and a straddle.

Parallel Can Be Slower

Wed, 13 Feb 2013 17:05:00 +0000

I found a nice finite difference scheme, where the solving part can be parallelized on 2 processors at each time-step

I was a bit surprised to notice that the parallelized algorithm ran in some cases twice slower than the same algorithm not parallelized. I tried ForkJoinPool, ThreadPoolExecutor, my one notify/wait based parallelization. All resulted in similar performance compared to just calling thread1.run() and thread2.run() directly.

I am still a bit puzzled by the results. Increasing the time of the task by increasing the number of discretization points does not really improve the parallelization. The task is relatively fast to perform and is repeated many (around of 1000) times, so synchronized around 1000 times, which is likely why parallelization is not great on it: synchronization overhead reaps any benefit of the parallelization. But I expected better. Using a Thread pool of 1 thread is also much slower than calling run() twice (and fortunately slower than the pool of 2 threads).

Scala is Mad (part 2)

Wed, 13 Feb 2013 16:20:00 +0000

I still did not abandon Scala despite my previous post, mainly because I have already quite a bit of code, and am too lazy to port it. Furthermore the issues I detailed were not serious enough to motivate a switch. But these days I am more and more fed up with Scala, especially because of the Eclipse plugin. I tried the newer, the beta, and the older, the stable, the conclusion is the same. It’s welcome but:code completion is not great compared to Java. For example one does not seem to be able to see the constructor parameters, or the method parameters can not be automatically populated.the plugin makes Eclipse very slow. Everything seems at least 3-5x slower. On the fly compilation is also much much slower than Java’s.It’s nice to type less, but if overall writing is slower because of the above issues, it does not help. Beside curiosity of a new language features, I don’t see any point in Scala today, even if some of the ideas are interesting. I am sure it will be forgotten/abandoned in a couple of years. Today, if I would try a new language, I would give Google Go a try: I don’t think another big language can make it/be useful on the JVM (beside a scripting kind of language, like JavaScript or Jython).Google Go focuses on the right problem: concurrency. It also is not constrained to JVM limitation (on the other side one can not use a Java library - but open source stuff is usually not too difficult to port from one language to another). It has one of the fastest compilers. It makes interesting practical choices: no inheritance.

From OpenSuse to Ubuntu 13.04

Fri, 01 Feb 2013 18:22:00 +0000

In my Linux quest, I changed distribution again on my home desktop, from OpenSuse 11.1 with KDE to Ubuntu 13.04 (not yet released - so alpha) with Unity. Why?

KDE was crashing a bit too often for my taste (more than once a week). Although I enjoyed the KDE environment.
Not easy to transfer files to my Android 4.2 phone. Ubuntu 13.04 is not fully there yet, but is on its way.
zypper is a bit too specific for my taste. I would be ok with yum+rpm or apt-get, but another tool just for a single distribution, not really.
Plus I’m just curious what’s next for Ubuntu, and Linux makes it very simple to change distributions, and reinstall applications with the same settings. So it’s never a big task to change distribution.
I somehow like how Ubuntu feels, not sure what it is exactly, maybe the Debian roots.

When people say OpenSuse is rock solid, I don’t have that impression, at least on the desktop. It might have been true in the past. But in the past, most distros were very stable. I never remember having big stability issues until, maybe, late 2010. In the early 2000s, a laptop would work very well with Linux, suspend included. I remember that my Dell Inspiron 8200 was working perfectly with Mandrake and WindowMaker. Nowadays, it never seem to work that well, but is just ok: Optimus comes to mind (especially with external screen), suspend problems, wifi (not anymore).

Productivity Zero

Thu, 24 Jan 2013 22:03:00 +0000

Sometimes it feels like big companies try to enforce the Productivity Zero rule.

Here is a guide:

involve as many team as possible. It will help ensuring endless discussions about who is doing what, and then how do I interface with them. This is most (in)efficient when team managers interact and are not very technically competent. One consequence is that nobody is fully responsible/accountable, which helps reinforce the productivity zero.
meetings, meetings and meetings. FUD (Fear Uncertainty and Doubt) is king here. By spreading FUD, there will be more and more meetings. Even if the “project” is actually not a real project, but amounts to 10 lines of code, it is possible to have many meetings around it, over the span of several months (because everybody is always busy with other tasks). Another strategy is to use vocabulary, talk about technical or functional parts the others don’t understand. Some people are masters are talking technical to functional people and vice versa.
multiply by 20, not 2. It is surprisingly easy to tell upper management something is going to take 3 months, when, in fact, it can be done in 3 days. This is a bit like bargaining in South East Asia: it’s always amazing to find out how much you can push the price down (or how much they push it up).
hire as many well paid managers and product specialists as you can, and make sure they know nothing about the product or the functional parts but are very good at playing the political game without any content. Those people often manage to stay a long time, a real talent.

Better Finite Difference Boundaries with a Tridiagonal Solver

Thu, 10 Jan 2013 19:19:00 +0000

In Pricing Financial Instruments - The Finite Difference Method, Tavella and Randall explain that boundary conditions using a higher order discretization (for example their “BC2” boundary condition) can not be solved in one pass with a simple tridiagonal solver, and suggest the use of SOR or some conjugate gradient based solver.

It is actually very simple to reduce the system to a tridiagonal system. The more advanced boundary conditions only use 3 adjacent values, just 1 value makes it non tridiagonal, the one in bold is the following matrix representation

Non Central Chi Squared Distribution in Java or Scala

Thu, 03 Jan 2013 17:21:00 +0000

I was looking for an implementation of the non central chi squared distribution function in Java, in order to price bond options under the Cox Ingersoll Ross (CIR) model and compare to a finite difference implementation. It turned out it was not so easy to find existing code for that in a library. I would have imagined that Apache common maths would do this but it does not.

OpenGamma has a not too bad implementation. It relies on Apache commons maths for the Gamma function implementation. I looked at what was there in C/C++. There is some old fortran based ports with plenty of goto statements. There is also a nice implementation in the Boost library. It turns out it is quite easy to port it to Java. One still needs a Gamma function implementation, I looked at Boost implementation of it and it turns out to be very similar to the Apache commons maths one (which is surprisingly not too object oriented and therefore quite fast - maybe they ported it from Boost or from a common source).

The Boost implementation seems much more robust in general thanks to:

The use of complimentary distribution function when the value is over 0.5. One drawback is that there is only one implementation of this, the Benton and Krishnamoorthy one, which is a bit slower than Ding's method.
Reverts to Benton and Krishnamoorthy method in high non-centrality cases. Benton and Krishnamoorthy is always very accurate, while Fraser (used by OpenGamma) is not very precise in general.

It is interesting to note that both implementations of Ding method are wildly different, Boost implementation has better performance and is simpler (I measured that my Java port is around 50% faster than OpenGamma implementation).

If only it was simple to commit to open-source projects while working for a software company...

Finite Difference Approximation of Derivatives

Fri, 21 Dec 2012 12:12:00 +0000

A while ago, someone asked me to reference him in a paper of mine because I used formulas of a finite difference approximation of a derivative on a non uniform grid. I was shocked as those formula are very widespread (in countless papers, courses and books) and not far off elementary mathematics.

There are however some interesting old papers on the technique. Usually people approximate the first derivative by the central approximation of second order:

A Discontinuity

Wed, 12 Dec 2012 20:59:00 +0000

I am comparing various finite difference schemes on simple problems and am currently stumbling upon a strange discontinuity at the boundary for some of the schemes (Crank-Nicolson, Rannacher, and TR-BDF2) when I plot an American Put Option Gamma using a log grid. It actually is more pronounced with some values of the strike, not all. The amplitude oscillates with the strike. And it does not happen on a European Put, so it's not a boundary approximation error in the code. It might well be due to the nature of the scheme as schemes based on implicit Euler work (maybe monotonicity preservation is important). This appears on this graph around S=350.

Update December 13, 2012: after a close look at what was happening. It was after all a boundary issue. It's more visible on the American because the Gamma is more spread out. But I reproduced it on a European as well.

Scala is Mad

Wed, 12 Dec 2012 16:07:00 +0000

I spent quick a bit of time to figure out why something that is usually simple to do in Java did not work in Scala: Arrays and ArrayLists with generics.

For some technical reason (type erasure at the JVM level), Array sometimes need a parameter with a ClassManifest !?! a generic type like [T :< Point : ClassManifest] need to be declared instead of simply [T :< Point].

And then the quickSort method somehow does not work if invoked on a generic... like quickSort(points) where points: Array[T]. I could not figure out yet how to do this one, I just casted to points.asInstanceOf[Array[Point]], quite ugly.

In contrast I did not even have to think much to write the Java equivalent. Generics in Scala, while having a nice syntax, are just crazy. This is something that goes beyond generics. Some of the Scala library and syntax is nice, but overall, the IDE integration is still very buggy, and productivity is not higher.

Update Dec 12 2012: here is the actual code (this is kept close to the Java equivalent on purpose):

object Point {
  def sortAndRemoveIdenticalPoints[T <: Point : ClassManifest](points : Array[T]) : Array[T] = {
      Sorting.quickSort(points.asInstanceOf[Array[Point]])
      val l = new ArrayBuffer[T](points.length)
      var previous = points(0)
      l += points(0)
      for (i <- 1 until points.length) {
        if(math.abs(points(i).value - previous.value)< Epsilon.MACHINE_EPSILON_SQRT) {
          l += points(i)
        }
      }
      return l.toArray
    }
    return points
  }
}

class Point(val value: Double, val isMiddle: Boolean) extends Ordered[Point] {
  def compare(that: Point): Int = {
    return math.signum(this.value - that.value).toInt
  }
}

In Java one can just use Arrays.sort(points) if points is a T[]. And the method can work with a subclass of Point.

Local Volatility Delta & Dynamic

Thu, 29 Nov 2012 12:30:00 +0000

This will be a very technical post, I am not sure that it will be very understandable by people not familiar with the implied volatility surface.

Something one notices when computing an option price under local volatility using a PDE solver, is how different is the Delta from the standard Black-Scholes Delta, even though the price will be very close for a Vanilla option. In deed, the Finite difference grid will have a different local volatility at each point and the Delta will take into account a change in local volatility as well.

But this finite-difference grid Delta is also different from a standard numerical Delta where one just move the initial spot up and down, and takes the difference of computed prices. The numerical Delta will eventually include a change in implied volatility, depending if the surface is sticky-strike (vol will stay constant) or sticky-delta (vol will change). So the numerical Delta produced with a sticky-strike surface will be the same as the standard Black-Scholes Delta. In reality, what happens is that the local volatility is different when the spot moves up, if we recompute it: it is not static. The finite difference solver computes Delta with a static local volatility. If we call twice the finite difference solver with a different initial spot, we will reproduce the correct Delta, that takes into account the dynamic of the implied volatility surface.

Here how different it can be if the delta is computed from the grid (static local volatility) or numerically (dynamic local volatility) on an exotic trade:

This is often why people assume the local volatility model is wrong, not consistent. It is wrong if we consider the local volatility surface as static to compute hedges.

OpenSuse 12.2

Wed, 14 Nov 2012 10:12:00 +0000

After too many upgrades of Ubuntu, and switching from Gnome to KDE and back, my Ubuntu system became behaving strangely in KDE: authorization issues, frequent crashes, pulseaudio and ardour problems. I decided to give another try to OpenSuse, as Linux makes it easy to switch system without losing too much time reinstalling the useful applications.

It’s been only a few days, but I am pleasantly surprised with OpenSuse. It feels more polished than Kubuntu. I could not point out to a specific feature, but so far I have not had to fiddle with any configuration file, everything works well out of the box. Somehow Kubuntu always felt flaky, read to break at any moment, while OpenSuse feels solid. But they should consider changing the default font settings in KDE to take advantage properly of antialiasing and pretty fonts (it’s only a few clicks away, but still the default is not the prettiest).

GPU computing in Finance

Mon, 15 Oct 2012 16:14:00 +0000

Very interesting presentation from Murex about their GPU computing. Some points were:

GPU demand for mostly exotics pricing and greeks
Local vol main model for EQD exotics. Local vol calibrated via PDE approach.
Markov functional model becoming main model for IRD.
Use of local regression instead of Longstaff Schwartz (or worse CVA like sim of sim).
philox RNG from DE Shaw. But the presenter does not seem to know RNGs very well (recommended Brownian Bridge for Mersenne Twister!).
An important advantage of GPU is latency. Grid computing only improves throughput but not latency. GPU improves both.

http://nvidia.fullviewmedia.com/gtc2010/0923-a7-2032.html

Pretty Fonts in Chrome with Linux

Wed, 12 Sep 2012 18:06:00 +0000

It's a bit incredible, but in 2012, some linux distros (like Fedora, or Kubuntu) still have trouble to have pretty fonts everywhere. I found a nice tip initially for Google Chrome but that seems to improve more than Chrome: create ~/.fonts.conf with the following:

[match target="font"]
[edit name="autohint" mode="assign"]
[bool]true[/bool]
[/edit]
[edit name="hinting" mode="assign"]
[bool]true[/bool]
[/edit]
[edit mode="assign" name="hintstyle"]
[const]hintslight[/const]
[/edit]
[/match]

replace [ and ] with brackets < and >

Update from 2013 - This can be done system wide, see http://chasethedevil.blogspot.com/2013/08/better-fonts-in-fedora-than-in-ubuntu.html

Fedora 17 vs Ubuntu 12.04

Wed, 12 Sep 2012 18:03:00 +0000

I had the bad idea to upgrade to the beta Ubuntu 12.10. Something awfully broke in the upgrade. After too much struggling with apt-get & dpkg, I decided to install Fedora 17.

Strangely Fedora feels much faster than Ubuntu 12.04 (the boot time especially). Yum seems also faster than apt-get, especially the update part. Also while the Unity dock is not bad (better than gnome shell dock), the Unity dash thing just makes me crazy, the gnome shell activities, while close, are much easier to use.

But it needs a bit more steps to install, although I had no problem to install nvidia drivers, Oracle 11g xe, Java, MP3, nice fonts, nice icons thanks to the guides here and there. SSD Trim instructions are the same (basically use "discard" instead of "default" in /etc/fstab). I have the most troubles with Oracle: somehow the start script does not work and I currently log in as "oracle" and start /u01/app/oracle/product/11.2.0/xe/config/scripts/startdb.sh from there (after having added the proper .bashrc for this user)

I even managed the peculiarities of my laptop a similar way as ubuntu: I want Nvidia card when plugged in to a monitor (to be able to use it) and Intel card when not (to be able to use the LCD screen). My solution is to use the boot screen (in reality one just need to restart X11): this amounted to add a "hdmi" in "/etc/grub.d/40_custom" and creating a link to "/etc/rc3.d/S10DriverSelect" the following script (a dirty hack):

#!/bin/sh
if grep -q hdmi /proc/cmdline
then
if [ -d /usr/lib64/xorg/modules.bak/extensions/nvidia ];
then
cp -f /etc/X11/xorg.conf.hdmi /etc/X11/xorg.conf
mv /usr/lib64/xorg/modules.bak/extensions/nvidia /usr/lib64/xorg/modules/extensions/
fi
else
if [ -d /usr/lib64/xorg/modules/extensions/nvidia ];
then
cp -f /etc/X11/xorg.conf.intel /etc/X11/xorg.conf
mv /usr/lib64/xorg/modules/extensions/nvidia /usr/lib64/xorg/modules.bak/extensions/
fi
fi

Linux distros are really becoming closer in terms of configuration, LSB has made great progress.

Edit from November 29: After a few weeks, I noticed that the system was quite unstable unfortunately. As a result, I moved back to Ubuntu on my laptop. I am running OpenSuse on my home computer. 1 year later, I am back to Fedora 19, 20 on my desktop - no stability issue, I prefer Gnome over KDE.

Binary Voting

Fri, 07 Sep 2012 17:21:00 +0000

How many reports have you had to fill up with a number of stars to choose? How much useless time is spent on figuring the this number just because it is always very ambiguous?

Some blogger wrote an interesting entry on Why I Hate Five Stars Reviews. Basically he advocates binary voting instead via like/dislike. Maybe a ternary system via like/dislike/don't care would be ok too.

One coworker used to advocate the same for a similar reason: people reading those reports only pay attention to the extremes: the 5 stars or the 0 stars. So if you want to have a voice, you need to express it via 5 or 0, nothing in between.

Moving The Needle

Tue, 21 Aug 2012 21:23:00 +0000

These days the expression “move the needle” is popular where I work. I did not know it was an expression of Steve Jobs.

“The company starts valuing the great salesmen, because they’re the ones who can move the needle on revenues, not the product engineers and designers. So the salespeople end up running the company… Then the product guys don’t matter so much, and a lot of them just turn off. It happened at Apple when John Sculley came in, which was my fault, and it happened when Ballmer took over at Microsoft. Apple was lucky and it rebounded, but I don’t think anything will change at Microsoft as long as Ballmer is running it.”

When solar panels don't work

Tue, 21 Aug 2012 21:14:00 +0000

I thought I would add another word about keyboard trends. A coworker has bought the Logitech K750, the one with solar panels to recharge the battery. This keyboard has excellent reviews on many websites, or even on Amazon. I somehow always found the idea a bit strange, it looked like the old solar panel calculators that used to be trendy when I was in primary school.

Well after maybe 6 months of use, he needs to change the battery! It sounds like the solar panels were just a marketing plot after all.

Excel Bulk Entry of Jira using Apache HttpClient & POI

Tue, 31 Jul 2012 15:40:00 +0000

Where I work, I have to regularly enter my time in JIRA using their crappy portlet interface. Because of French regulations and bad design, one can enter time for at most 1 day at a time. This is very annoying especially to enter vacation days. I decided to spend some time (took me around 2 hours - I thought it would be much more) to enter the time from a local Excel spreadsheet (via with OpenOffice), and use Java to populate JIRA.

First I had to find out what where the relevant requests. Firefox has several extensions for that, but I found Tamper Data to be the easiest to work with (hint: use copy/paste in the Tamper Data window to get the full request in a nice format).

Apache HttpClient provides an easy way to do HTTP requests and handles cookies almost automatically in Java. Here is the login phase:

List<NameValuePair> formparams = new ArrayList<NameValuePair>();
formparams.add(new BasicNameValuePair("os_username", "mouse@thecat"));
formparams.add(new BasicNameValuePair("os_password", "DEADDEAD"));
UrlEncodedFormEntity entity = new UrlEncodedFormEntity(formparams, "UTF-8");
HttpPost httppost = new HttpPost("https://jira.calypso.com/rest/gadget/1.0/login");
httppost.setEntity(entity);
DefaultHttpClient httpclient = new DefaultHttpClient();
CookieStore cookieStore = new BasicCookieStore();
httpclient.setCookieStore(cookieStore);
ResponseHandler<byte[]> handler = new ResponseHandler[]>() {
	public byte[] handleResponse(HttpResponse response)
			throws ClientProtocolException, IOException {
		System.out.println("<-" + response.getStatusLine());
		HttpEntity entity = response.getEntity();
		if (entity != null) {
			return EntityUtils.toByteArray(entity);
		} else {
			return null;
		}
	}
};
System.out.println("->" + httppost.getURI());
byte[] response = httpclient.execute(httppost, handler);

Then a request to our JIRA portlet looks like:

formparams = new ArrayList<NameValuePair>();
formparams.add(new BasicNameValuePair("inline", "true"));
formparams.add(new BasicNameValuePair("decorator", "dialog"));
formparams.add(new BasicNameValuePair("startDate", startDate));
formparams.add(new BasicNameValuePair("timeLogged", timeLogged));
formparams.add(new BasicNameValuePair("id", id));
formparams.add(new BasicNameValuePair("adjustEstimate", "auto"));
entity = new UrlEncodedFormEntity(formparams, "UTF-8");
httppost = new HttpPost("https://jira.calypso.com/secure/CreateWorklog.jspa");
httppost.addHeader("Referer", "https://jira.calypso.com/browse/"+ jiraCAL);
httppost.setEntity(entity);
System.out.println("->" + httppost.getURI());
response = httpclient.execute(httppost, handler);

Parsing Excel with Apache POI is a bit annoying, but I kept fixed conventions to make things simple:

InputStream inp = new FileInputStream(file);
HSSFWorkbook wb = new HSSFWorkbook(new POIFSFileSystem(inp));
List list = new ArrayList();
HSSFSheet sheet = wb.getSheetAt(0);
boolean isEmpty = false;
int i = 0;
while (!isEmpty) {
	HSSFRow row = sheet.getRow(i);
	if (row == null) { isEmpty=true; break;}
	
	HSSFCell dateCell = row.getCell(0);
	HSSFCell calCell = row.getCell(1);
	HSSFCell idCell = row.getCell(2);
	HSSFCell percentCell = row.getCell(3);
	if (dateCell == null) {
		isEmpty = true;
	} else if (dateCell.getCellType() == HSSFCell.CELL_TYPE_NUMERIC && calCell != null){
		TimeLine timeLine = new TimeLine();
		timeLine.date = HSSFDateUtil.getJavaDate(dateCell.getNumericCellValue());
		if (timeLine.date.after(startDate)
				&& timeLine.date.before(endDate)) {
			timeLine.jiraCAL = calCell.getStringCellValue();
			if (timeLine.jiraCAL != null && timeLine.jiraCAL.length() > 0) {
				timeLine.id = Integer.toString((int)idCell.getNumericCellValue());
				timeLine.percent =  Integer.toString((int)percentCell.getNumericCellValue());
				list.add(timeLine);
			}
		}
	}
	i++;
}

Obviously, this is not clean code, the goal was only to do something quick and dirty to solve my immediate problem.

Keyboard Porn

Mon, 30 Jul 2012 10:49:00 +0000

I have been browsing the web, looking for a nice computer keyboard. Programming is a big part of my job, a comfortable keyboard is therefore important to my daily life.

MS Natural keyboard

I have some nostalgia for the original Microsoft natural keyboard, the white one with standard home and end keys. When it came out it looked revolutionary. I remember really improving my typing speed on it. The only minor annoyance was the heavy and loud space bar. It’s sad that MS does not make those in azerty anymore (I know qwerty is better for programming - I used it while I was working in the US, but it’s just too annoying to have a different layout from 95% of the people around). The new MS ergonomic model just looks ugly and scary with all those extra keys.

Adaptive Quadrature for Pricing European Option with Heston

Mon, 25 Jun 2012 12:50:00 +0000

The Quantlib code to evaluate the Heston integral for European options is quite nice. It proposes Kahl & Jaeckel method as well as Gatheral method for the complex logarithm. It also contains expansions where it matters so that the resulting code is very robust. One minor issue is that it does not integrate both parts at the same time, and also does not propose Attari method for the Heston integral that is supposed to be more stable.

I was surprised to find out that out of the money, short expiry options seemed badly mispriced. In the end I discovered it was just that it required sometimes more than 3500 function evaluations to have an accuracy of 1e-6.

As this sounds a bit crazy, I thought that Jaeckel log transform was the culprit. In reality, it turned out that it was Gauss Lobatto Gander & Gautschi implementation. I tried the simplest algorithm in Espelid improved algorithms: modsim, an adaptive extrapolated Simpson method, and it was 4x faster for the same accuracy. That plus the fact that it worked out of the box (translated to Java) on my problem was impressive.

Jaeckel log transform (to change the interval from 0,+inf to 0,1) works well, and seems to offer a slight speedup (10% to 15%) for around ATM options, mid to long term for the same accuracy. Unfortunately, it can also slow down by up to 50% the convergence for more OTM options or shorter expiries. So I am not so sure about its interest vs just cutting off the integration at phi=200.

Gnome Shell more stable than Unity on Ubuntu 12.04

Thu, 14 Jun 2012 12:01:00 +0000

Regularly, the unity dock made some applications inaccessible: clicking on the app icon did not show or start the app anymore, a very annoying bug. This is quite incredible given that this version of Ubuntu is supposed to be long term support. So I decided to give one more chance to Gnome Shell. Installing it on Ubuntu 12.04 is simple with this guide.

To my surprise it is very stable so far. Earlier Gnome Shell versions were not as stable. After installing various extensions (dock especially) it is as usable as Unity for my needs. It seems more responsive as well. I am not really into the Unity new features like HUD. It sounds to me like Ubuntu is making a mistake with Unity compared to Gnome Shell.

To make an old extension support latest Gnome Shell version, it is sometimes necessary to update the extension metadata with what's given by gnome-shell --version. For the weather extension you can just edit using gedit:

sudo gedit /usr/share/gnome-shell/extensions/weather@gnome-shell-extensions.gnome.org/metadata.json

John Carmack on Parallelism

Fri, 27 Apr 2012 06:40:00 +0000

This is the interesting bit

“Modify some of your utility object code to return new copies instead of self-mutating, and try throwing const in front of practically every non-iterator variable you use”.

http://www.altdevblogaday.com/2012/04/26/functional-programming-in-c/

Google Galaxy Nexus Sound Quality Is Great

Tue, 27 Mar 2012 00:38:00 +0000

Many people are not enthusiastic of this phone sound if you read silly forums. They are wrong! the sound coming out of this thin phone is amazing, at least with high quality headphones. I find the akg q601 incredible with it: much much better than with the old ipod nano or the cowon i7.

In general most complaints i have read about the phone were wrong. The battery is ok, the size is great.

Why primitive arrays matter in Java

Wed, 29 Feb 2012 10:01:00 +0000

In the past, I have seen that one could greatly improve performance of some Monte-Carlo simulation by using as much as possible double[][] instead of arrays of objects.

It was interesting to read this blog post explaining why that happens: it is all about memory access.

Scala Again

Mon, 06 Feb 2012 17:52:00 +0000

I am trying Scala again. Last time, several years ago, I played around with it as a web tool, combining it with a Servlet Runner like Tomcat. This time, I play around with it for some quantitative finance experiments.

Why Scala? It still seem the most advanced alternative to Java on the JVM, and the mix of functional programming and OO programming is interesting. Furthermore it goes quite far as it ships with its own library. I was curious to see if I could express some things better with Scala.

Here are my first impressions after a week:

I like the object keyword. It avoids the messy singleton pattern, or the classes with many static methods. I think it makes things much cleaner to not use static at all but distinguish between object & class.
I like the Array[Double], and especially ArrayBuffer[Double]. Finally we don't have to worry between the Double and double performance issues.
I was a bit annoyed by a(i) instead of a[i] but it makes sense. I wonder if there is a performance implication for arrays, hopefully not.
I like the real properties, automatic getter/setter: less boilerplate code, less getThis(), setThat(toto).
Very natural interaction with Java libraries.
I found a good use of case classes (to my surprise): typically an enum that can have some well defined parameters, and that you don't want to make a class (because it's not). My use case was to define boundaries of a spline.
I love the formatter in the scala (eclipse) IDE. Finally a formatter in eclipse that does not produce crap.

Now things I still need time to get used to:

member variable declared implicitly in the constructor. I first made the mistake (still?) to declare some variables twice.
I got hit by starting a line with a + instead of ending with a +. It is dangerous, but it certainly makes the code more consistent.
Performance impacts: I will need to take a look at the bytecode for some scala constructs to really understand the performance impact of some uses. For example I tend to use while loops instead of for comprehension after some scary post of the Twitter guys about for comprehension. But at first, it looks as fast as Java.
I wrote my code a bit fast. I am sure I could make use of more Scala features.
The scala IDE in eclipse 3.7.1 has known issues. I wish it was a bit more functional, but it's quite ok (search for references works, renaming works to some extent).
Scala unit tests: I used scala tests, but it seems a bit funny at first. Also I am not convinced by the syntax that avoid method names and prefer test("test name"). It makes it more difficult to browse the source code.

Some things they should consider:

Integrate directly a Log API. I just use SLF4J without any scala wrapper, but it feels like it should be part of the standard API (even if that did not work out so well for Sun).
Double.Epsilon is not the machine epsilon: very strange. I found out somewhere else there was the machine epsilon, don't remember where because I ended up just making a small object.
Unit tests should be part of the standard API.

Overall I found it quite exciting as there are definitely new ways to solve problems. It was a while since I had been excited with actual coding.

KDE 4.8 finally has a dock

Fri, 27 Jan 2012 13:38:00 +0000

KDE 4.8 finally has a dock: you just have to add the plasma icon tasks. Also the flexibility around ALT+TAB is welcome. With Krusader as file manager, Thunderbird and Firefox for email and web, it is becoming a real nice desktop, but it took a while since the very bad KDE 4.0 release.

It is easy to install under ubuntu 11.10 through the backports and seems very stable so far.

Something quite important is to tweak the fonts: use Déjà Vu Sans instead of Ubuntu fonts, use RGB subpixel rendering, use Crisp desktop effects. With those settings, KDE looks very nice. It's sad that they are not default in Kubuntu.

Update March 2013: It's been a while now that it is in the standard Ubuntu repositories and I believe installed by default, one has just to remove the task manager widget add the icon task widget:

One can also change the settings using a right click (I find useful not to highlight the windows) and it can look like:

List of companies where I have been an employee

Wed, 11 Jan 2012 02:14:00 +0000

Intern:

Siemens (Berlin)
IBM (Boeblingen)
Osram Sylvania (Beverly, MA)

Employee:

Cap Gemini (Paris) working for Alcatel
Silicomp (Paris) working for Alcatel Nextenso
C2labs / one 0 development (San Francisco, CA) working for Whenmobile, Sony Pictures, GoPix, Technorati.
Credit Agricole alternative (Paris)
Prima solutions (Paris)
Esearchvision (Paris)
Ulink (Paris)
Edifixio (Paris)
Horizon (Paris)
Darty (Paris)
Calypso (Paris)

Generating random numbers following a given discrete probability distribution

Mon, 09 Jan 2012 00:14:00 +0000

I have never really thought very much about generating random numbers according to a precise discrete distribution, for example to simulate an unfair dice.

In finance, we are generally interested in continuous distributions, where there is typically 2 ways:

the inverse transform (usually computed in a numerical way),
and the acceptance-rejection method, typically the ziggurat.

The inverse transform is often preferred, because it’s usable method for Quasi Monte-Carlo simulations while the acceptance rejection is not.I would have thought about the simple way to generate random numbers according to a discrete distribution as first described here. But establishing a link with Huffman encoding is brilliant. Some better performing alternative (unrelated to Huffman) is offered there.

Quant Interview & Education

Wed, 21 Dec 2011 17:37:00 +0000

Recently, I interviewed someone for a quant position. I was very surprised to find out that someone who did one of the best master in probabilities and finance in France could not solve a very basic probability problem:

This is accessible to someone with very little knowledge of probabilities

When I asked this problem around to co-workers (who have all at least a master in a scientific subject), very few could actually answer it properly. Most of the time, I suspect it is because they did not dedicate enough time to do it properly, and wanted to answer it too quickly.

It was more shocking that someone just out of school, with a major in probabilities could not answer that properly. It raises the question: what is all this education worth?

The results were not better as soon as the question was not exactly like what students in those masters are used to, like for example, this simple stochastic calculus question:

My opinion is that, today in our society, people study for too long. The ideal system for me would be one where people learn a lot in math/physics the first 2 years of university, and then have more freedom in their education, much like a doctorate.

We still offered the job to this person, because live problem solving is not the most important criteria. Other qualities like seriousness and motivation are much more valuable.

Gnome 3 not so crap after all

Wed, 30 Nov 2011 18:11:00 +0000

In a previous post, I was complaining how bad Gnome 3 was. Since I have installed a real dock: docky, it is now much more usable. I can easily switch / launch applications without an annoying full screen change.

In addition I found out that it had a good desktop search (tracker). The ALT+F2 also does some sort of completion, too bad it can not use tracker here as well.

So it looks like Gnome 3 + gnome-tweak-tool + docky makes a reasonably good desktop. XFCE does not really fit the bill for me: bad handling of sound, bad default applications, not so good integration with gnome application notifications.

Now if only I found a way to change this ugly big white scrollbar…

Good & Popular Algorithms are Simple

Thu, 17 Nov 2011 12:28:00 +0000

I recently tried to minimise a function according to some constraints. One popular method to minimise a function in several dimensions is Nelder-Mead Simplex. It is quite simple, so simple that I programmed it in Java in 1h30, including a small design and a test. It helped that the original paper from Nelder-Mead is very clear:

However the main issue is that it works only for unconstrained problems. Nelder and Mead suggested to add a penalty, but in practice this does not work so well. For constrained problems, there is an adaptation of the original idea by Box (incredible name for a constrained method) that he called the Complex method, a deliberate pun to the Nelder-Mead Simplex method. The basic idea is to reset the trial point near the fixed boundary if it goes outside. Now this took me much longer to program, as the paper is not as clear, even if the method is still relatively simple. But worst, after a day of deciphering the paper and programming the complex method, I find out that it does not works so well: it does not manage to minimise a simple multidimensional quadratic with a simple bound constraint: f(X)=sum(X)^2 with X >= 0 where X is an N-dimensional vector. In the end, I don't want to work with such a simple function, but it is a good simple test to see if the method is really working or not. Here is how the function looks with N=2:

And if we restrict to x,y >= 0 it becomes:

I suspected an error in my program, so I decided to try with scilab, that has also the Box method as part of their neldermead_search functionality. Scilab also failed to compute the minimum in 8 dimensions of my simple quadratic.

I tried various settings, without ever obtaining a decent result (I expect to find a value near 0).

There is another algorithm that can find a global minimum, also very popular: the Differential Evolution. At first, being a genetic algorithm, you would think it would be complicated to write. But no, the main loop is around 20 lines.

Those 20 lines are a bit more difficult than Nelder-Mead, but still, when I saw that, I understood that "this is a classic algorithm". And it does work with constraints easily. How to do this is explained well in K. Price book "Differential Evolution", and takes only a few lines of code. Here is the result I got in dimension 29:
dim=29 min=1.2601854176573729E-12 genMax=412 x=[3.340096901536317E-8, 8.889252404343621E-8, 7.163904251807348E-10, 9.71847877381699E-9, 2.7423324674150668E-8, 2.4022269439114537E-9, 1.7336434478718816E-11, 7.244238163901778E-9, 1.0013136274729337E-8, 7.412154679865083E-9, 5.4694460144807974E-9, 2.3682413086417524E-9, 4.241739250073559E-7, 4.821920889534676E-10, 2.115396281722523E-9, 8.750883007882899E-8, 2.512011485133975E-9, 4.811507109129279E-9, 1.0752997894113096E-7, 5.120475258343548E-9, 8.404448964497456E-9, 4.1062290228305595E-9, 1.7030766521603753E-8, 5.589430643552073E-9, 8.237098544820173E-10, 3.5796523161196554E-9, 5.186299547466997E-9, 2.430326342762937E-7, 5.493850433494286E-9]

It works well, although there is quite a bit of parameters. I noticed that the strategy was especially important.

Gnome 3, Unity, Crap

Wed, 17 Aug 2011 19:09:00 +0000

After the upgrate to Ubuntu 11.04 I was directly on Unity. Having a dock on the left side is nice, but unfortunately, it has various bugs which makes it sometimes annoying. Also the menu on top like Mac Os X is not a bad idea, but it breaks many apps (for example Picasa). Then there is the scrollbar insanity, it’s almost impossible to click on to scroll, and again breaks in some apps (for example Eclipse). Fortunately one can disable all of that and go back to the old Gnome 2.

Carmack & GPGPU programming

Sun, 14 Aug 2011 10:20:00 +0000

Finally someone who shares the same opinion on the current state of GPGPU programming.

John Carmack: On the other hand, we have converted all of our offline processing stuff to ray tracing. For years, the back-end MegaTexture generation for Rage was done with… we had a GPGPU cluster with NVIDIA cards and it was such a huge pain to keep. It was an amazing pain where one system would be having heat problems and would be behaving weird even though we thought they had identical drivers. Something would always be wrong with render farm 12, and whenever we wanted to put in new features it was like “Okay, writing new fragment programs to go into this.” Now, granted I did this just when CUDA was in its infancy. If I did re-implement it with OpenCL or CUDA we wouldn’t have some of these problems, but when I converted all these over to ray tracing there was a number of things that got a lot better. Things that we deal with, for example shadows and reflections that have to be approximated, and were so used to doing with rasterization… we sometimes forget how big of hacks these are. To be able to say I really just want that ray, and tell me what it hit; not do a projection with feathering shadowed edges and whatever the heck else we’re doing there, so much of the code got so much easier. If it’s a choice of… now that we have these awesome multi-core x86 CPUs where we can get 24 threads in commodity boxes… it’s true that one GPU card can do more ray tracing than one 24 thread x86 box, but it’s not multiples more and if it’s just a matter of buying more $2000 boxes, it makes the development, maintenance, and upkeep much better. While everyone in high performance computing is all “rah-rah” GPUs right now, I’ve come full circle back around to saying the fact that we can get massive amounts of x86 cores and threads… it wont win on FLOPS/watt or FLOPS/volume, but in terms of results per developer hour it is much, much better.

exp(y*log(x)) Much Faster than Math.pow(x,y)

Fri, 08 Apr 2011 23:03:00 +0000

Today I found out that replacing Math.pow(x,y) by Math.exp(yMath.log(x))* made me gain 50% performance in my program. Of course, both x and y are double in my case. I find this quite surprising, I expected better from Math.pow.

Best Headphones Ever?

Wed, 06 Apr 2011 13:08:00 +0000

I just miraculously received some AKG Q701 headphones. I never tried really high end headphones before those. I was used to my Sennheiser HD555, which I thought were quite good already.

I always thought there was not much difference between let’s say a $100 headphone and a $500 one, and that only real audiophiles would be able notice the difference: people like Jake who can distinguish artefacts in 320kbps mp3, but not people like me. I was wrong. Those headphones make a real difference, everything seems very clear, and it doesn’t feel like listening music through headphones at all, it is much closer to a good hi-fi stereo sound on the ear.

SIMD and Mersenne-Twister

Sat, 05 Feb 2011 13:18:00 +0000

Since 2007, there is a new kind of Mersenne-Twister (MT) that exploits SIMD architecture, the SFMT. The Mersenne-Twister has set quite a standard in random number generation for Monte-Carlo simulations, even though it has flaws.

I was wondering if SFMT improved the performance over MT for a Java implementation. There is actually on the same page a decent Java port of the original algorithm. When I ran it, it ended up slower by more than 20% than the classical Mersenne-Twister (32-bit) on a 64-bit JDK 1.6.0.23 for Windows.

XORWOW L'ecuyer TestU01 Results

Wed, 12 Jan 2011 20:26:00 +0000

Nvidia uses XorWow random number generator in its CURAND library. It is a simple and fast random number generator with a reasonably long period. It can also be parallelized relatively easily. Nvidia suggests it passes L'Ecuyer TestU01, but is not very explicit about it. So I've decided to see how it performed on TestU01.

I found very simple to test a new random number generator on TestU01, the documentation is great and the examples helpful. Basically there is just a simple C file to create & compile & run.

XORWOW passes the SmallCrush battery, and according to Marsaglia, it also passes DieHard. But it fails on 1 test in Crush and 3 in BigCrush. By comparison, Mersenne-Twister fails on 2 tests in Crush and 2 in BigCrush. Here are the results of the failures:

========= Summary results of Crush =========

Version:          TestU01 1.2.3
Generator:        Xorwow
Number of statistics: 144
Total CPU time:   00:50:15.92
The following tests gave p-values outside [0.001, 0.9990]:
(eps means a value < 1.0e-300):
(eps1 means a value < 1.0e-15):

       Test                          p-value
----------------------------------------------
72 LinearComp, r = 29             1 - eps1
----------------------------------------------
All other tests were passed

========= Summary results of BigCrush =========

Version:          TestU01 1.2.3
Generator:        Xorwow
Number of statistics: 160
Total CPU time:   06:12:38.79
The following tests gave p-values outside [0.001, 0.9990]:
(eps means a value < 1.0e-300):
(eps1 means a value < 1.0e-15):

       Test                          p-value
----------------------------------------------
7 CollisionOver, t = 7            1.6e-6
27 SimpPoker, r = 27                eps
81 LinearComp, r = 29             1 - eps1
----------------------------------------------
All other tests were passed

The CUDA Performance Myth

Mon, 03 Jan 2011 16:07:00 +0000

There is an interesting article on how to generate efficiently the inverse of the normal cumulative distribution on the GPU. This is useful for Monte-Carlo simulations based on normally distributed variables.

Another result of the paper is a method (breakless algorithm) to compute it apparently faster than the very good Wichura’s AS241 algorithm on the CPU as well keeping a similar precision. The key is to avoid branches (if-then) at the cost of not avoiding log() calls. As the algorithm is very simple, I decided to give it a try in Java.

Firefox 4 Is Great

Wed, 22 Dec 2010 16:03:00 +0000

I temporarily abandonned Firefox for Chrome/Chromium. I am now back at using Firefox as Firefox 4 is as fast or faster than Chrome and seems more stable, especially under linux. Also it does not send anything to Google and there is bookmark sync independently of Google.

I am impressed that Mozilla managed to improve Firefox that much.

Another Look at Java Matrix Libraries

Mon, 29 Nov 2010 12:45:00 +0000

A while ago, I was already looking for a good Java Matrix library, complaining that there does not seem any real good one where development is still active: the 2 best ones are in my opinion Jama and Colt.

Recently I tried to price options via RBF (radial basis functions) based on TR-BDF2 time stepping. This is a problem where one needs to do a few matrix multiplications and inverses (or better, LU solve) in a loop. The size of the matrix is typically 50x50 to 100x100, and one can loop between 10 and 1000 times.

Java enum Is Evil

Thu, 12 Aug 2010 17:32:00 +0000

Before Java 1.5, I never really complained about the lack of enum keyword. Sure the old enum via class pattern was a bit verbose at first (N.B.: Java 1.5 enums can also be verbose once you start adding methods to them). But more importantly, you would often use the table lookup pattern in combination.

The problem with Java 1.5 enum is that it is not Object-Oriented. You can't extend an enum, you can't add an element in an existing enum. Many will say "but that's what enum is for, a static list of things". In my experience, the list of things often changes with time, or needs to be extended at one point. Furthermore, most people (including me when I am very lazy) end up writing switch statements on enum values. Enum promotes bad programming practices.

Think twice about using enum, this is often not what you want.

A Very Interesting Feature of Scala

Sat, 07 Aug 2010 12:35:00 +0000

I tried Scala a few years ago. There are several good ideas in it, but I found the language to be a bit too complicated to master. But I recently stubbled upon a paper on Scala generics that might change my mind about using Scala.

Scala Generics used to work in a similar way as Java Generics: via type erasure. One main reason is compatibility with Java, another is that C++ like templates make the code base blow up. Scala Generics offered some additional behavior (the variance/covariance notion). C++ templates, however, have some very interesting aspects: one is that everything is done at compile time, the other is performance. If the generics are involved in any kind of computation intensive task, all the Java type conversion will create a significant overhead.

Now Scala has @specialized (since Scala 2.8). Annotating a generic type with @specialized will generate code. One has the choice to accept the performance penalty or to get all the performance but accept the code blow up. I think this is very useful.

If you read the paper you will see that the performance implications of this are not always small.

UPDATE: I thank the readers for pointing that this work only with primitive types to avoid autoboxing. It is still valuable but less than I first thought.

Street Fighting Mathematics Book

Wed, 28 Jul 2010 14:25:00 +0000

The MIT has a downloadable book on basic mathematics: Street Fighting Mathematics. I liked the part focused on the geometrical approach. It reminded me of the early greek mathematics.

Overall it does look like a very American approach to Maths: answering a multiple choices questions test by elimination. But it is still an interesting book.

Bye Bye Firefox

Wed, 21 Jul 2010 15:30:00 +0000

I have been a long user of Firefox, mostly thanks to the adblock extension. But recently, Firefox decided to change the way arrows work on the web pages, they don’t make the page scroll anymore. Meanwhile Chrome has now a good adblock plugin (that filters ads on load, not after load like it use to be) and is really much much faster than Firefox. So there is no more reason not to use it.

Diffusion Limited Aggregation Applet

Wed, 09 Jun 2010 14:05:00 +0000

Yes, I wrote an applet. I know it is very 1990s but, amazingly, it still does the job quite well. Ok, next time I should really use Flash to do this.

The Applet simulates Diffusion Limited Aggregation as described in Chaos And Fractals from Peitgen, Juergens, and Saupe. It represents ions randomly wandering around (in a Brownian motion) until they are caught by an attractive force in electrochemical deposition experiment. This kind of phenomenon occurs at all scales, for example it happens in the distribution of galaxies. You can play around with the applet at http://31416.appspot.com/dla.vm

Java & 3D Surface

Fri, 18 Dec 2009 16:07:00 +0000

I have been looking all around the web for a Java library that can draw a simple 3D surface. And I did not find any. Most charting library, including the well known JFreeChart, can only draw 2D charts.

I am quite shocked that something that has been in Excel for 15 years is still not available in Java. And it’s not easy to make your own.

double[][] Is Fine

Thu, 26 Nov 2009 14:51:00 +0000

In my previous post, I suggest that keeping a double[] performs better than keeping a double[][] if you do matrix multiplications and other operations.

This is actually not true. I benchmarked 3 libraries, Colt (uses double[]), Apache Commons Math (uses double[][]) and Jama (uses double[][] cleverly). At first it looks like Jama has a similar performance as Colt (they avoid [][] slow access by a clever algorithm). But once hotspot hits, the difference is crazy and Jama becomes the fastest (Far ahead).

JDK 1.6.0 Linux 1000x1000 matrix multiplication on Intel Q6600
loop index	Colt	Commons Math	Jama
1	11.880748	24.455125	9.828977
2	11.874975	24.265102	9.848916
3	9.772616	14.374153	9.826572
4	9.759679	14.368105	2.655915
5	9.799622	15.238928	2.649129
6	9.780556	14.741863	2.668104
7	9.72831	15.509909	2.646811
8	9.79838	15.724348	2.646069
9	9.726143	15.988762	2.646052
10	9.784505	15.121782	2.644572
	We don't include matrix construction time, and fetching the result. Only the multiplication is taken into account.

The difference is less pronounced on smaller matrices, but still there. Jama looks very good in this simple test case. In more real scenarios, the difference is not so obvious. For example Commons Math SVD is faster than Jama one.

The Pain of Java Matrix Libraries

Thu, 26 Nov 2009 09:17:00 +0000

Looking for a good Java Matrix (and actually also math) library, I was a bit surprised to find out there does not seem to be any really serious one still maintained.

Sure, there is Apache Commons Math, but it is still changing a lot, and it is not very performance optimized yet, while it has been active for several years already. There is also Java3D, it does Matrix through GMatrix, but not much linear algebra and if you look at their implementation, it is very basic, not performance oriented.

Java Calendar Is Broken On JVM Upgrade

Wed, 19 Aug 2009 13:47:00 +0000

We ran into an interesting issue with TimeZone and Dates. If you print the same date on different JVMs, it might show a different printed date.

The reason behind this is the daylight saving time conventions. An old JVM won't necessarily have the same daylight saving time for a given TimeZone than a latest JVM, and therefore will interpret the date differently.

Here is the output of a very simple program on 2 different JVMs. The number is the long number used to represent the date. I only use SimpleDateFormat with different TimeZone:

JVM 1.5.0_12
loading date=Sat Jul 18 06:59:36 CEST 2009, number=1247893176505
using formatter in EST: 7/17/09 11:59 PM
using formatter in Asia/Singapore: 7/18/09 12:59 PM

JVM 1.5.0_20
loading date=Sat Jul 18 06:59:36 CEST 2009, number=1247893176505
using formatter in EST: 7/18/09 12:59 AM
using formatter in Asia/Singapore: 7/18/09 12:59 PM

The source code:


public class DateBug {       
   
    private static String FILE_NAME = "datebug.txt";
   
    public static void load() throws IOException {
        FileReader fr = new FileReader(FILE_NAME);
        BufferedReader br = new BufferedReader(fr);
        String l = br.readLine();
        br.close();
        long time = new Long(l);
        Date d = new Date(time);
        System.out.println("loading date="+d+", number="+d.getTime());
        SimpleDateFormat formatter = new SimpleDateFormat();
        formatter.setTimeZone(TimeZone.getTimeZone("EST"));
        System.out.println("using formatter in EST: "+formatter.format(d));
        formatter.setTimeZone(TimeZone.getTimeZone("Asia/Singapore"));
        System.out.println("using formatter in Asia/Singapore: "+formatter.format(d));
    }
   
    public static void saveNew() throws IOException {
        Calendar c = Calendar.getInstance(TimeZone.getTimeZone("EST"));
        c.set(2009, 06, 17, 23, 59);
        Date d = c.getTime();
        System.out.println("saving date="+d+", number="+d.getTime());
        SimpleDateFormat formatter = new SimpleDateFormat();
        formatter.setTimeZone(TimeZone.getTimeZone("EST"));
        System.out.println("using formatter in EST: "+formatter.format(d));
        formatter.setTimeZone(TimeZone.getTimeZone("Asia/Singapore"));
        System.out.println("using formatter in Asia/Singapore: "+formatter.format(d));
       
        FileWriter fw = new FileWriter(FILE_NAME);
        PrintWriter pw = new PrintWriter(fw);
        pw.println(d.getTime());
        pw.close();
    }
   
    public static void main(String[] args) throws IOException {
        System.out.println("JVM "+System.getProperty("java.version"));
        if (args.length == 1) {
            if (args[0].equals("save")) {
                saveNew();
            }
        } else {
            load();
        }
    }

What does this mean? This means that if you entered in a GUI in the first JVM a particular date & time using EST time zone. This will change when you read back in the second JVM.
This suggests that if you want to keep the same dates, you are better off saving in UTC where daylight saving time is not used and trick DateFormat. But I have to say this looks quite ugly.

Implicit Finite Differences Method For Pricing Barrier Option

Mon, 17 Aug 2009 12:04:00 +0000

While trying to price a simple knock down and out barrier option, I encountered several difficulties I did not expect with the implicit finite differences method. The explicit method has less issues with barrier options pricing. I will show here what the tricky parts are and why explicit seems simpler in this case.

The full article is here (pdf) or here (html) (the later is not very well formatted).

Algorithms used in this article can be found at http://code.google.com/p/javamc/

Pulseaudio Nightmares - Pure ALSA to the Rescue

Sat, 27 Jun 2009 18:19:00 +0000

In the latest stable Ubuntu (9.04), pulseaudio still does not work reliably on my hardware (intel HDA with digital SPDIF out). I upgraded to 9.10 and had even more problems. Sound always worked on boot, but often broke down after a while. And I could not find easy ways to make it work other than rebooting... Killing/restarting pulseaudio, looking at the processes using snd did not work.

One thing works wonderfully, pure ALSA. To have multilple apps sharing ALSA, I just use dmix. As I use digital out, there is no mixer, but ALSA can provide one through softvol. It works really well. ALSA is already not that simple to configure/setup properly, but with pulseaudio on top, welcome to your worst configuration nightmares.

Here is the .asoundrc I use:

 
pcm.amix {
   type dmix
   ipc_key 50557
   slave {
       pcm "hw:0,1"
       period_time 0
       period_size 1024
       buffer_size 8192
   }
   bindings {
       0 0
       1 1
   }
}

pcm.softvol {
    type            softvol
    slave {
        pcm         "amix"      #redirect the output to dmix (instead of "hw:0,0")
    }
    control {
        name        "PCM"       #override the PCM slider to set the softvol volume level globally
        card        0
    }
}


pcm.!default {
    type            plug
    slave.pcm       "softvol"   #make use of softvol
}

Java int Overflow Behavior

Mon, 15 Jun 2009 16:15:00 +0000

A coworker recently asked me if there was a guaranteed behavior in the case of int overflow. He gave the specific example on:

can we rely that int x = Integer.MAX_VALUE + 1 is the same for every JVM on any platform?

I thought the answer would be easy to find in the Java specifications document. But I was wrong. It is not clearly defined.

I found a trick that suggests this behavior is indeed standard and will stay the same, it is related to type casting. Java guarantees that the cast of a long just truncates the long to int precision. Therefore if

long l = Integer.MAX_VALUE;
l = l +1;
x = (int)l;

Static Fields and Inheritance

Mon, 15 Jun 2009 16:02:00 +0000

Someone asked me recently to find out the real reason why the code from this thread fails. This is a fairly bad code, and not even a very good way to point out the problem. But the question is nonetheless interesting.

class Toto extends TotoParent{

 final static Toto a = new Toto ("a");

 public Toto(String a){
  super(a);
 }
}

import java.util.ArrayList;
import java.util.List;

public abstract class TotoParent {

 static List list = new ArrayList();
 
 public TotoParent(String a) {
  list.add(a);
 }

 protected static List get() {
  return list;
  
 }
}

import org.junit.Test;
import static org.junit.Assert.*;

public class TotoTest {

 @Test
 public void testGet(){
  assertEquals(1, Toto.get().size());
 }
}

I am quite used to static initialization, and would have answered the same as the first answer in the thread:
"Get is static and associated with TotoParent, so that is the same as calling TotoParent.get().size()". I would have even thought that the compiler would compile the call Toto.get() to TotoParent.get(). But running javap, you can see it is still compiled as TotoParent.get(). So there is still a lookup done. This is why the first answer is actually not that correct.

The important bit here is that Toto is never initialized, even if we call Toto.get(). The java specs (invaluable reference) explains clearly that calling a static method not declared in the class does not initialize the class.

Calling Toto.get() is not exactly the same as calling TotoParent.get().
If TotoParent.get() called another TotoSuperParent.get():
Toto.get() -> TotoParent.get() -> TotoSuperParent.get()
We compile then later we change to make TotoParent have a specific implementation of get(). Toto will then be automatically aware of it, without even recompiling it.

http://java.sun.com/docs/books/jls/third_edition/html/execution.html
paragraph 12.4.1

Benchmarking Languages Is Difficult

Wed, 03 Jun 2009 16:53:00 +0000

I often looked at the famous computer languages shootout for fun. Recently I noticed they had the infamous thread ring test. I posted not very long ago several blog entries about it showing how silly this test was.

Looking at the existing Java implementation for the test I decided to try to submit the tricky one using a pool of thread, and pooling message processing rather creating 1 thread per node. To my surprise, it was accepted without questions and I did have the best score for a Java program for a while. Shortly after someone else copied my program and got rid of various stuff not useful for the particular benchmark (breaking the interesting part of the design) and got accepted as well with of course a better result.

I decided to see if I could make an even more silly program - tailored for the test only. I managed to be orders of magnitude faster - 1 thread, no synchronization, everything processed in a FIFO (linkedlist) queue. This is actually a standard way to reimplement recursion. But I was honest enough not to hide that I consider that kind of program to cheat the test and got my entry in the "interesting alternatives".

In reality there is no difference in the "cheating" between my new program and the program that got accepted in the official list, they both cheat by using only 1 thread and process everything 1 by 1. There is not 1 thread per node in any of the program, and they can avoid any concurrency issues. One "looks" better because it uses a pool of 503 threads (but really use only 1 or 2 threads) and the other does not hide its use of 1 thread for processing. But this is not evident to people accepting the programs.

When I look at the haskell code, I can not really tell if it is creating 503 threads in the language or a pool or ..., you have to know each language quite well and sometimes it is not that easy to define what cheating is. Therefore this kind of benchmark is a bit disappointing. One should force the use of the same algorithm. But can you do so (a functional language won't use the same algo as a procedural one)?

Cholesky & Jakarta Commons Math

Fri, 15 May 2009 19:01:00 +0000

In Finance, Cholesky is a useful way to decompose Matrix. It is not so simple to find a BSD licensed code using cholesky (most of them are GPL like this one). There is one in Apache Commons Maths library, which is a very interesting library. However for performance, it is still not very practical for some things like Cholesky.

Looking at the source one can easily understand why. I did a small (many people will say not representative 1 million loop test) and finds out:

Hull American Option Price Fallacies

Fri, 15 May 2009 16:01:00 +0000

Hull says American put is best exercised immediately and american call is optimal at expiry like a european. Is this really true?

At first it seems really clever and model show clearly this. But if we change the market assumptions only a tiny bit, everything falls down.

I could not detail everything in a blog post so I created a static web page about it. Everything was produced in Java using algorithm found in popular books and graphs through JFreeChart.

On Quasi Random Numbers - MersenneTwister vs Sobol precision in Asian Option Pricing

Tue, 05 May 2009 15:02:00 +0000

While starting a side project that does Monte Carlo pricing in Java (http://code.google.com/p/javamc/ - nothing yet there I am waiting for Mercurial repository support), I wondered what was the importance of quasi random numbers versus more regular pseudo random numbers in Monte Carlo simulations.

This brought me to read more carefully several books about Monte Carlo and Finance (Haug Option Pricing, Sobol Primer on Monte Carlo, and Glasserman Monte Carlo Methods in Finance Engineering). I had quite a hard time to understand why the dimension of the quasi random generator was so important to price an asian option. Intuitively I thought the averaging points of an asian option were all on the same path, so they should be using the same random generator. This is very wrong as one does not care about the path in the first place but just in simulating each point in the average (using the regular black and scholes hypothesis). Finding the estimation for the average on the given points forces to use independent random generators for each point, because we want to approximate the estimation by the sum over those random points for each point.

There is another simple argument to explain why independence of the random generators is so important. If we use the same generator for each point, then each point will move exactly the same way at each simulation. The average of those point will therefore behave exactly the same way as if there was only 1 point using the same generator. And we don't price an asian anymore but just a regular vanilla option.

Using a pseudo random generator, one does not see the problem of dimension, because we can create N independent dimensions by just taking numbers N by N on a pseudo random generator. So effectively having 1 or N dimensions is the same on a pseudo random generator.

Still I wrote a small test to see if a 1D quasi random generator was so bad when simulating N dimensions (taking values N by N on the quasi random generator). Here are the results:

MersenneTwister vs MersenneTwister on 10D asian:
14:43:51,111 INFO MonteCarloSimulationTest:114 - 867970000 -- expPrice=0.978958644504466
14:43:51,428 INFO MonteCarloSimulationTest:120 - 314619000 -- expPrice=0.9733220318545934
14:43:51,430 INFO MonteCarloSimulationTest:122 - relative difference=-0.005757763804951897
can be as high as 2%

Sobol vs MersenneTwister on 10D asian:
14:48:46,909 INFO MonteCarloSimulationTest:115 - 980209000 -- expPrice=0.9895032774079221
14:48:47,345 INFO MonteCarloSimulationTest:121 - 433685000 -- expPrice=0.9790264042895171
14:48:47,348 INFO MonteCarloSimulationTest:123 - relative difference=-0.010588012548932534
about 1% it is actually bounded by MersenneTwister precision.

Sobol vs Sobol1D on 10D asian:
14:47:08,614 INFO MonteCarloSimulationTest:115 - 717444000 -- expPrice=0.8810736428068913
14:47:08,925 INFO MonteCarloSimulationTest:121 - 308499000 -- expPrice=0.9791449305055208
14:47:08,927 INFO MonteCarloSimulationTest:123 - relative difference=0.11130884290920073
about 10% and stays that way even when increasing number of simulations.

Using an asian rate with 10 points, we see that Sobol1D will always give a very bad estimate, no matter the number of simulations. While Sobol used properly will give (much) better precision for less iterations. So even though there is the word random in quasi random, the numbers are very far from being random or even behaving like random numbers. It helped me to read about Van der Corput and Halton numbers to really understand quasi random numbers.

Java Logging Still Crap in 2009

Fri, 24 Apr 2009 15:01:00 +0000

When java logging API was first introduced in JDK 1.4 in 2002, it caused quite a lot a fuss around, with everybody asking “Why did not they just include Log4j instead of creating their own bastard child?”.

I remember having looked at it very shortly before continuing using Log4j on all projects I have been involved with.

Today, while doing a very small project, I tried once more to use java logging. The main reason is that I was lazy to add a dependency to one more jar for this small project. While trying I found out that:

Bachelier vs. Black

Mon, 23 Mar 2009 17:58:00 +0000

Black and Scholes gives a strange result for the price of a binary option under high volatility. You will learn here how to simulate a stock price evolution using Java, and how to show it using JFreeChart library. It starts with more complex concepts (don't be afraid) and goes done towards simpler things.

I could not write all that in a blog format, so I created a old HTML page about it here and a PDF version.

Linux Audio State = Miserable

Thu, 19 Mar 2009 12:39:00 +0000

There are lots of programs for playing MP3 under linux, a few dealing decently with big libraries. But when you start looking for a program that does crossfade well and manage big libraries easily - there is nothing.

Rhythmbox does some crossfade, but crashes when you move manually in the song. Audacious does some crossfade but regularly crashes with crossfade plugin.

The real alternative are AIMP2 or Foobar2000 in Wine. It is quite incredible that you can have good solid crossfade in wine and not natively in Linux.

Senior Developers Team Productivity X4 (from MS Research Paper)

Tue, 10 Feb 2009 10:44:00 +0000

There is a very interesting MS Research paper about test driven development (TDD). It is one of the only real study about it that I know of. The paper conclusions from experiments over 4 TDD teams vs 4 traditional teams is:

"TDD seems to be applicable in various domains and can significantly reduce the defect density of developed software without significant productivity reduction of the development team"

Their data gives also other interesting results:

An experienced team (5 people over 10 years + 2 people under 5 years) : 155KLOC C# code (+60 test).

A junior team (3 people under 10 years + 6 people under 5 years): 41 KLOC Java code (+28 test).

If you do the ratio of KLOC/man month, you have the following graph:

I know this is very far from scientific evidence and more like astrology, but still, the most conservative ratio for senior/junior is 4.23!

The End Of Rings Around Plain Java - A Better Concurrency Test

Thu, 15 Jan 2009 15:54:00 +0000

In my previous post, I was wondering why single thread was faster. D Andreou gave the correct explanation: as we send only 1 start message and as each node only send 1 message to the next one, there is always only 1 message being processed. So the test is optimum on 1 thread. It does not make much sense to make a multithreading benchmark on a problem that is fundamentally single threaded.

Object Oriented Analysis And Design with Applications Book Review

Thu, 08 Jan 2009 17:24:00 +0000

A while ago, I had a comment from someone implying I knew nothing about OO programming because I had not mentioned (and therefore read) Object Oriented Analysis And Design with Applications from G. Booch. I was intrigued by such a silly comment and decided to look at this book that was considered as the bible of OOP.

Well, I don’t find it that good! But I don’t find the bible particularly good either. I like B. Meyer Object Oriented Software Construction book much more, because it is more practical, more in touch with realities while pointing at the real important problems like:

Running Rings Around Plain Java - The Killer Code

Thu, 08 Jan 2009 13:21:00 +0000

I wrote my previous post too fast. I found a very simple change that increases the speed x6!

The idea is too process messages in a ThreadPoolExecutor. As my Nodes are Runnable, I just needed to initialize a common ThreadPoolExecutor, and in a sendMessage, execute the runnable each time.

Here is the full code: public class OptimizedRing {

    private ExecutorService executor;

    public static void main(String[] args) throws Exception {
        OptimizedRing ring = new OptimizedRing();
        RingNode node = ring.startRing(Integer.parseInt(args[0]));
        node.sendMessage(new StartMessage());
    }

    public RingNode startRing(int n) {
        RingNode[] nodes = spawnNodes(n, startTimer());
        connectNodes(n, nodes);
        return nodes[0];
    }

    private Timer startTimer() {
        Timer timer = new Timer();
        new Thread(timer).start();
        return timer;
    }

    private RingNode[] spawnNodes(int n, final Timer timer) {
        System.out.println("constructing nodes");
        long start = System.currentTimeMillis();
        executor = Executors.newFixedThreadPool(4);
        RingNode[] nodes = new RingNode[n+1];
        for (int i = 0; i < n ; i++) {
            nodes[i] = new RingNode(i, timer, null);
        }
        long end = System.currentTimeMillis();
        System.out.println("Took "+(end-start)+"ms to construct "+n+" nodes");
        return nodes;
    }

    private void connectNodes(int n, RingNode[] nodes) {
        System.out.println("connecting nodes");
        nodes[n] = nodes[0];
        for (int i=0; i<n; i++) {
            nodes[i].connect(nodes[i+1]);
        }
    }

    interface Message {
        String getType();
    }

    private static class StartMessage implements Message {
        public String getType() {
            return "START";
        }
    }

    private static class StopMessage implements Message {
        public String getType() {
            return "STOP";
        }
    }

    private static class CancelMessage implements Message {
        public String getType() {
            return "CANCEL";
        }
    }

    private static class TokenMessage implements Message {
        private int nodeId;
        private int value;

        public TokenMessage(int nodeId, int value) {
            this.nodeId = nodeId;
            this.value = value;
        }

        public String getType() {
            return "TOKEN";
        }
    }

    private class RingNode implements Runnable {
        private int nodeId;
        private Timer timer;
        private RingNode nextNode;
        private BlockingQueue<Message> queue = new LinkedBlockingQueue<Message>();
        private boolean isActive = false;

        public RingNode(int id, Timer timer, RingNode nextNode) {
            nodeId = id;
            this.timer = timer;
            this.nextNode = nextNode;
        }

        public void connect(RingNode node) {
            nextNode = node;
            isActive = true;
        }

        public void sendMessage(Message m) {
            queue.add(m);
            executor.execute(this);
        }

        public void run() {
            if (isActive) {
                try {
                    Message m = queue.take();
                    if (m instanceof StartMessage) {
                        log("Starting messages");
                        timer.sendMessage(m);
                        nextNode.sendMessage(new TokenMessage(nodeId, 0));
                    } else if (m instanceof StopMessage) {
                        log("Stopping");
                        nextNode.sendMessage(m);
                        isActive = false;
                        //
                    } else if (m instanceof TokenMessage) {
                        if (((TokenMessage)m).nodeId == nodeId) {
                            int nextValue = ((TokenMessage)m).value + 1;
                            if (nextValue % 10000 == 0) {
                                log("Around ring "+nextValue+" times");
                            }
                            if (nextValue == 1000000) {
                                timer.sendMessage(new StopMessage());
                                timer.sendMessage(new CancelMessage());
                                nextNode.sendMessage(new StopMessage());
                                isActive = false;
                            } else {
                                nextNode.sendMessage(new TokenMessage(nodeId, nextValue));
                            }

                        } else {
                            nextNode.sendMessage(m);
                        }
                    }

                } catch (InterruptedException ie) {
                    ie.printStackTrace();
                }
            }
        }

        public void log(String s) {
            System.out.println(System.currentTimeMillis()+" "+nodeId+": "+s);
        }

    }

    private static class Timer implements Runnable {
        private BlockingQueue<Message> queue = new LinkedBlockingQueue<Message>();
        private boolean timing = false;
        private long startTime;

        public void sendMessage(Message m) {
            //we don't need to change this implementation as timer is rarely called
            queue.add(m);
        }

        public void run() {
            while (true) {
                Message m;
                try {
                    m = queue.take();
                    if (m instanceof StartMessage) {
                        startTime = System.currentTimeMillis();
                        timing = true;
                    } else if (m instanceof StopMessage) {
                        long end = System.currentTimeMillis();
                        System.out.println("Start="+startTime+" Stop="+end+" Elapsed="+(end-startTime));
                        timing = false;
                    } else if (m instanceof CancelMessage) {
                        break;
                    }
                } catch (InterruptedException e) {
                    e.printStackTrace();
                }
            }
        }
    }
}

Code	Spawn	Send 100M messages
Scala Actors	15ms	270104ms
SimpleRing	11ms	493073ms
OptimizedRing (4 threads)	6ms	84727ms
OptimizedRing (5+ threads)	5ms	62593ms
OptimizedRing (1 thread)	5ms	60660ms

I finally saw my 4 cores used! Max multithreaded throughput is achieved at 5 threads. However 1 thread is faster. Is this related to memory bandwith limit?

Now I am left wondering if actors are really that important if one can achieve much higher throughput using plain Java and very simple concepts (BlockingQueue, ThreadPoolExecutor). Worse, this test is actually faster with only 1 thread...

Running rings around plain Java

Thu, 08 Jan 2009 12:26:00 +0000

Alex Miller has a very interesting test of Actors. He finds out Scala performance is relatively low compared to Erlang, and Kilim is very near Erlang. But Kilim code is the most difficult to read in the lot.

I thought it would be simple to just do the same test in plain Java. I wrote the code for it duplicating the scala logic using Threads instead of Actors.

public class SimpleRing {
    public static void main(String[] args) throws Exception {
        SimpleRing ring = new SimpleRing();
        RingNode node = ring.startRing(Integer.parseInt(args[0]));
        node.sendMessage(new StartMessage());
    }

    public RingNode startRing(int n) {
        RingNode[] nodes = spawnNodes(n, startTimer());
        connectNodes(n, nodes);
        return nodes[0];
    }

    private Timer startTimer() {
        Timer timer = new Timer();
        new Thread(timer).start();
        return timer;
    }


    private RingNode[] spawnNodes(int n, final Timer timer) {
        System.out.println("constructing nodes");
        long start = System.currentTimeMillis();
        RingNode[] nodes = new RingNode[n+1];

        for (int i = 0; i < n ; i++) {
            nodes[i] = new RingNode(i, timer, null);
            new Thread(nodes[i]).start(); //later use pool
        }

        long end = System.currentTimeMillis();
        System.out.println("Took "+(end-start)+"ms to construct "+n+" nodes");

        return nodes;
    }


    private void connectNodes(int n, RingNode[] nodes) {
        System.out.println("connecting nodes");
        nodes[n] = nodes[0];

        for (int i=0; i<n; i++) {
            nodes[i].connect(nodes[i+1]);
        }
    }


    interface Message {
        String getType();
    }


    private static class StartMessage implements Message {
        public String getType() {
            return "START";
        }
    }

    private static class StopMessage implements Message {
        public String getType() {
            return "STOP";            
        }
    }


    private static class CancelMessage implements Message {
        public String getType() {
            return "CANCEL";
        }
    }


    private static class TokenMessage implements Message {
        private int nodeId;
        private int value;

        public TokenMessage(int nodeId, int value) {
            this.nodeId = nodeId;
            this.value = value;
        }

        public String getType() {
            return "TOKEN";
        }
    }


    private static class RingNode implements Runnable {
        private int nodeId;
        private Timer timer;
        private RingNode nextNode;
        private BlockingQueue<Message> queue = new LinkedBlockingQueue<Message>();

        public RingNode(int id, Timer timer, RingNode nextNode) {
            nodeId = id;
            this.timer = timer;
            this.nextNode = nextNode;                        
        }


        public void connect(RingNode node) {
            nextNode = node;
        }


        public void sendMessage(Message m) {
            queue.add(m);
        }


        public void run() {
            while (true) {
                try {
                    Message m = queue.take();
                    if (m instanceof StartMessage) {
                        log("Starting messages");
                        timer.sendMessage(m);
                        nextNode.sendMessage(new TokenMessage(nodeId, 0));
                    } else if (m instanceof StopMessage) {
                        log("Stopping");
                        nextNode.sendMessage(m);
                        break;
                    } else if (m instanceof TokenMessage) {
                        if (((TokenMessage)m).nodeId == nodeId) {
                            int nextValue = ((TokenMessage)m).value + 1;
                            if (nextValue % 10000 == 0) {
                                log("Around ring "+nextValue+" times");
                            }
                            if (nextValue == 1000000) {
                                timer.sendMessage(new StopMessage());
                                timer.sendMessage(new CancelMessage());
                                nextNode.sendMessage(new StopMessage());
                                break;
                            } else {
                                nextNode.sendMessage(new TokenMessage(nodeId, nextValue));
                            }
                        } else {
                            nextNode.sendMessage(m);
                        }
                    }

                } catch (InterruptedException ie) {
                    ie.printStackTrace();
                }
            }
        }


        public void log(String s) {
            System.out.println(System.currentTimeMillis()+" "+nodeId+": "+s);
        }
    }


    private static class Timer implements Runnable {
        private BlockingQueue<Message> queue = new LinkedBlockingQueue<Message>();
        private boolean timing = false;
        private long startTime;


        public void sendMessage(Message m) {
            queue.add(m);
        }


        public void run() {
            while (true) {
                Message m;
                try {
                    m = queue.take();
                    if (m instanceof StartMessage) {
                        startTime = System.currentTimeMillis();
                        timing = true;
                    } else if (m instanceof StopMessage) {
                        long end = System.currentTimeMillis();
                        System.out.println("Start="+startTime+" Stop="+end+" Elapsed="+(end-startTime));
                        timing = false;                                        
                    } else if (m instanceof CancelMessage) {
                        break;
                    }
                } catch (InterruptedException e) {
                    e.printStackTrace();
                }
            }
        }
    }
}

I was a bit surprised by the result. It was slow and only 1 thread was really active at one time. This is why the test is particularly good. It is not trivial to reproduce the functionality in plain Java in an effective manner. It really shows how the concept of Actors can be useful.

More Double Troubles

Tue, 06 Jan 2009 12:12:00 +0000

We saw in a previous entry how one has to be careful with Double.NaN. Today we will see how regular double can cause problems. By the way the NaN issue was not Java specific and this issue is also general in different programming languages.

A coworker was shocked that in Java (I was a bit surprised he saw that only today, but it is true it can be surprising that such a simple thing does not work as expected):

408.16 - 40.82 = 367.34000000000003

In C, this would lead to the same result. This is all due to the binary represention of double numbers. Using the formula 2^(exponent)*1.mantissa where mantissa is on 52 bits, we have

408.16 decomposition:

exponent = 256. Then 408.16/256 = 1.594375 = 1 + 0x9828F5C28F5C28F5C28F5C... * 2^-52
We round to 52 bits, the mantissa is 0x9828F5C28F5C3 = 2676827028518339.
As a decimal, the internal value is (2676827028518339/2^52+1) * 256 = 408.1600000000000250111042987555265426635742

40.82 decomposition:

exponent = 32. Then 40.82/32= 1.275625 = 1 + 0x468F5C28F5C28F5C...*2^-52
Rounded to 52 bits, the mantissa is 0x468F5C28F5C29 = 1241304647293993
As a decimal, the internal value is (1241304647293993/2^52 + 1)*32 = 40.8200000000000002842170943040400743484497

The difference in decimal becomes 367.34000000000002472... which becomes 367.34000000000003 when represented in binary (to convince yourself you can apply the same technique).

The Solution

One solution to this problem is to use java.math.BigDecimal which stores a number as 2 integers, one for the digits, one for the exponent power of 10 (and not 2).
The correct code would become:

BigDecimal value = BigDecimal.valueOf(408.16).subtract(BigDecimal.valueOf(40.82));

value would then be 367.34.

But BigDecimal has also many potential for bugs. For example, you should never use the constructor taking a double but always the one taking a String.

new BigDecimal(408.16) = 408.16000000000002501110429875552654266357421875

This is because of the binary representation of 408.16 as a double. 408.16 is only an approximation of 408.16!

Another trick with BigDecimal is not to use equals(Object) but compareTo(Object) because 408.160 is not equal to 408.16 using equals.

Why Could not They Make it Work With Double?

If you were too lazy to follow the steps of the explanation. There is a simpler explanation. Imagine the representation of a number in base 3 with 2 "digits". Let's imagine 1/3 is represented as 0.1 (this is a very simple number representation) 1/3+1/3+1/3 becomes 0.1+0.1+0.1 = 1.0 (in base 3) = 1.0 if we convert to base 10. Now in base 10, 1/3 can only be represented as 0.3, so 1/3+1/3+1/3 = 0.3+0.3+0.3 = 0.9 <> 1.0.
So BigDecimal is only interesting to handle ... decimals! In the enterprise world, this should be most apps. It is a bit sad it appeared so late in the JDK. It should really be a primitive type.

Java Programmers Are Idiots?

Wed, 17 Dec 2008 20:16:00 +0000

My brother just sent me a funny quote. I don’t know if it is true or not:

entwickeln Sie lieber überzeugende Lösungen anstatt viele Stunden mit Coding zu verbringen? Ist Ihnen die Produktivität Ihres Teams wichtig?

Mark Driver, VP Research von Gartner, kommentierte kürzlich

“Here’s a simple equation. In terms of mental fortitude… 1 Smalltalk developer = 2.5 C++ developers 1 C++ developer = 1.5 Java developers”.

You don’t need german to understand. Of course it can not be true. How can anyone measure mental fortitude? And how does it related with productivity is another issue.

Double.NaN Is Evil

Tue, 09 Dec 2008 16:23:00 +0000

I don't know what Sun had in mind when creating Double.NaN number. It is very inintuitive to use. I am sure every single developer out there fell in the trap of trying to find out if a double was NaN or not using:

Double.NaN == myDouble

This does not work (I don't know the real reason why), one has to use:

Double.isNaN(myDouble)

Not intuitive!

Grails Spring Union Not Surprising

Thu, 27 Nov 2008 17:18:00 +0000

Looking out at some old post. I found out I was not far from the truth in January 2008 when I stated:“In 2008 the Ruby On Rails mentality will continue to prevail. In the Java world, Grails is the most likely to benefit from it. (…) It could also be something based around Spring as their current MVC solution is not very good and very old fashioned."I don’t think I will be right with the provocative Java is dead. A post recently titled No Future In Java makes some good points about where the future of Java still is: the web applications. Grails is probably today the best contender in the Java world, far ahead from the others, and it leverages the Java developers. However I am not sure one can say that RIA is a fad, or that RIA will only be done in a super powerful browser in the future. Microsoft might be a game changer here. The real advantages of the browser application so far are: easy deployment (the killer argument IMHO), “simple” security. I would not be surprised if in the near future, Microsoft advertises a solution for RIA easily deployed, based on standard protocols (HTTP?), a bit like IBM does with Eclipse RIA, but much more ambitious.

Java Is Dead

Fri, 21 Nov 2008 20:24:00 +0000

These days, I have the feeling that Java is dead. Even if, or maybe because I have used Java so much in the past 10 years, I have this feeling.

In 1998 Java was revolutionary. It was a very simple to learn object oriented language with modern concepts and familiar syntax. Furthermore the standard library had neat features like internet networking and it could be integrated in the browser. All this at a time when the internet just started to be popular.

Why ArchLinux Is Better Than Ubuntu

Fri, 14 Nov 2008 10:41:00 +0000

It has been now a week since I have installed ArchLinux on my home computer. I daily use Ubuntu 8.10 at work.

Since the Ubuntu upgrade from 8.04 to 8.10 I have had problems with my Xorg settings. I just found out the nvidia-settings utility does not manage to save the configuration anymore. So I have to lookup on google and try to fix it. And that annoys me. That annoys me because the promess of Ubuntu is that everything works out of the box. In reality, you have to mess with the configuration as much as with ArchLinux.

KDE 4.1.3 (again) on ArchLinux

Sat, 08 Nov 2008 15:54:00 +0000

I tried another silly thing with Linux, ArchLinux. The setup is quite rough as you have to edit many config files manually. But if you know a bit your way around it takes only a few hours to have everything running well. The installation manual on the wiki is detailed enough to correct all eventual mistakes humans do.

I decided to try once more KDE 4 on it, as at first it was just a silly experiment: I was really not sure ArchLinux would be workable. In the end I am pleasantly surprised, KDE 4.1.3 is way way better than any other versions of KDE I have tried before. It is stable and quite pretty. It took the team a lot of time to get there but now I think KDE 4 is a very good window manager, pleasant to use.

Stupid Programmer Interviews

Wed, 17 Sep 2008 10:56:00 +0000

I have read a blog post a few days ago about someone thinking a good programmer interview question was:

How does a hash table work?

While it is a very interesting question, I doubt many programmers (even relatively good ones) can answer that question. If I look back and think of all the employees in all the companies I have known, I can count on one hand people that can answer that question. I can think of 3 or 4 I met in one company, and maybe another 1 or 2 in different companies. And I don’t think anyone would have been able to go deeper in the details like mentioning closed-addressed vs open-addressed possible implementations.

The Art of Multiprocessor Programming Book Review

Thu, 11 Sep 2008 19:32:00 +0000

I don’t remember why I started to subscribe to the Java concurrency-interest list. I find that overall, it is an excellent mailing list.

There was a post at one point about the Dante Inferno’s problem. It triggered my attention, so I decided to buy the book the post was referring to, The Art of Multiprocessor Programming by M Herlihy and N. Shavit.

The books starts with the basics, and is very didactic in its approach. I enjoyed to learn how locks work and how to build them almost out of nothing. The progression is good, starting with a half broken but simple lock and evolving to the more standard algorithm, like the Bakery Lock algorithm. The algorithms are extremely well explained. Later it explains the differences between spin locks (Bakery for example) and blocking locks, while presenting new algorithms for blocking locks.

OO Desillusion

Thu, 28 Aug 2008 17:00:00 +0000

I have read many inspiring books about object oriented programming. I find B. Meyer Object Oriented Software Construction one of the best in the lot. B. Meyer tries to explain in a progressive way why OO is better, by introducing it bit by bit. I have read different related design patterns book, the GoF one, Martin Fowler ones. I have been programming Java for about 10 years now. And yet, today, I feel unconvinced.

Linus vs OpenBSD

Thu, 17 Jul 2008 11:38:00 +0000

Today’s popular story is Linus Torvalds message to OpenBSD where he writes:

I think the OpenBSD crowd is a bunch of masturbating monkeys

Beside that provoking sentence he has a valid point. Security bugs should not be more important than other bugs. Too often, management and psychology encourage making security bugs a very important issue and security people VIPs. I have seen this over and over.

Suse 11 with KDE4

Thu, 17 Jul 2008 11:38:00 +0000

I was very disappointed by KDE 4.0 when I first tried it on an ubuntu machine. It was just unusable. I would not have even considered it as a beta.

I changed the system on my home laptop because Ubuntu with KDE 4 was there. I decided to go for something more roots. I had good memories of Gentoo when I tried it some years ago. Maybe I just became too old to appreciate it anymore. But after spending several hours on a Grub error 2 problem, seeing the live CD does not even use grub so you can’t use the CD to boot from hard drive and fix the problem easily, wondering if the problem was a problematic Grub version with my drive or not (I never had problems with Grub and other distros on that same machine before), I decided to install OpenSuse 11.

Java Compilation Performance / Processor Speed

Tue, 01 Jul 2008 19:50:00 +0000

I just found out my laptop was faster in default settings than my home desktop to compile a resonably sized project (5 min vs 6 min). I was surprised as I thought the disk in the desktop would make a big difference. The processor in my desktop is not that great (simple pentium e2180). My laptop has a 2ghz core2duo processor. In Ghz processor are of the same speed.

I tried to overclock my home pc to see what difference it could make, I made it run at 2.6Ghz instead of the standard 2Ghz. The compilation time dropped to 4.5 min.

Firefox 3

Wed, 18 Jun 2008 11:26:00 +0000

I tried Firefox 3 twice before, while it was in alpha and beta. I was not impressed at all, it seemed buggy (normal for alpha) and I found the new location bar behavior unintuitive. It did not seem that much faster either. So I always quickly went back to Firefox 2.

I tried again recently, funnily, because of a bad Kernel update. One day, after a kernel update, my laptop started to run only at a max of 800Mhz, I did not notice it immediately. I thought wow Firefox 2.0 is really slow with gmail, maybe I should try Firefox 3 again. I did and worked a few days without noticing the processor speed difference that much. Firefox 3 at 800Mhz is as fast as Firefox 2.0 at 2Ghz. It is only after struggling with very long compilation time related to my work that I thought something was wrong.

As usual with Linux troubles, solution are strange but not too hard to find out. I just had to add a processor.ignore_ppc=1 kernel parameter for the kernel to behave properly again.

Yesterday I installed Firefox 3 on a MacBook, and I was really impressed by its speed. I felt faster than Safari. And now I got used to the new location bar behavior, I would not go back (I think the behavior improved after the betas as well).

Another interesting browser these days, especially for bloggers, is Flock 2 (based on Firefox 3).

Trying Google AppEngine

Tue, 17 Jun 2008 19:26:00 +0000

I finally took some time to try Google AppEngine. It used to be easy to find free PHP hosting around 2000. It became a rarity. So writing small experiments for free on the web was difficult. Experiments are back thanks to Google with their AppEngine. Many aspects of it are quite interesting and show where they focus.

First it is all Python. It makes sense as I believe Guido v Rossum, Python creator, works for Google. Some people believe in a future Java application hosting. I don’t see any reason why it could not become a reality. Making something like AppEngine is a big task, changing implementation language is not. In the meantime, it is not an excuse not to try it, as the Python standard library is fairly rich and Google provides additional libraries on top of it.

Option, Futures and Other Derivatives Book Review

Mon, 16 Jun 2008 17:06:00 +0000

Option, Futures and Other Derivatives is by far the most popular book in finance. You will find it in every finance company, on many desks.

It is a very good introduction for people not familiar with standard financial products. This kind of book is unavoidable to understand the basis. It goes also beyond with the chapters on pricing and hedging. These 2 chapters make one understand many other book. If one understands the Black and Scholze formula, one can easily approach many other pricing formula, as in a way, there are all similar in their approach.

Structured Equity Derivatives Book Review

Mon, 16 Jun 2008 16:55:00 +0000

If one has to learn about equity derivatives, beside the classic Option, Futures and Other Derivatives from Hull, Structured Equity Derivatives by Harry M Kat is a must read.

His ideas are presented in a software developer friendly way, as his goal is to show how different equity derivatives products are behind the scenes, very similar.

I enjoyed the variety of exotic products presented and the very detailed way in which they are explained

Fedora Linux & Apple Bluetooth Keyboard

Thu, 05 Jun 2008 20:05:00 +0000

It took me a long time to have the Apple Bluetooth Keyboard (slim aluminium model) working well with Fedora 9. Thanks to the Ubuntu documentation, it is quite easy to establish a connection manually through hidd commands. It was unfortunately much more difficult to have it automatically recognized and not disconnected after a few minutes without use.

The following configuration should work with any other distro and probably other bluetooth keyboards as well.

2 configurations files need to be updated to make it work:

/etc/default/bluetooth should contain the following, with the mac address replaced by the one from your keyboard (hidd --search to see it).

BLUETOOTH_ENABLED=1

HIDD_ENABLED=1
HIDD_OPTIONS="--timeout 8 --connect 00:1D:4F:A7:15:CC --server"

/etc/bluetooth/hcid.conf should be appended with

device 00:1D:4F:A7:15:CC {
name "Apple Wireless Keyboard";
auth enable;
encrypt enable;
lm master;
}

The 2 tricky parameters are "lm master" (makes it connect automatically), and "--timeout 8" that makes it not disconnect contrary to what the option tells. Without the option the timeout of the keyboard is about 11 minutes. If the computer timeout is lower, the connection will be reestablished automatically, thus the value of 8.

Jun 16th update: I still had disconnections, the only way I found to get rid of them was to patch the kernel. I find surprising it was needed, I thought there would have been enough people complaining that the patch would have already been applied in Fedora kernel.

DecimalFormat Is Broken

Wed, 14 May 2008 18:19:00 +0000

A friend of mine recently noticed that the good old DecimalFormat class is "broken". If you try to parse a string that is not a number but is starting with a number, the DecimalFormat.parse will return what it managed to parse.

The correct behavior should be to throw a parse exception IMHO. Judging from an old post in the Sun bug tracker, The folks at Sun don't think it really is, they call the default mode of parsing the "lenient" mode. It accepts bad inputs. Then why throwing ParseException at all and why not return 0/NaN when the first character is not a number? Why accepting 1toto2 as a number and not toto2?

In reality it can really create unexpected problems. For example, in France, 0.1 is 0,1 because of the Locale conventions. If a user enters 0.1 in a French Locale, a method using DecimalFormat.parse will interpret it as 0 without throwing any exception.

Note that DateFormat does not have that problem, at one point Sun added setLenient flag to be able to be in non Lenient mode. It would be very simple to do it with DecimalFormat, I did it myself as an exercise. In DecimalFormat.subparse, the 2 last break statements should stop processing in lenient mode. Lines 1528 to 1531:
sawExponent = true;
}
break; // Whether we fail or succeed, we exit this loop
}
else {
break;
}
become:

sawExponent = true;
} else {
if (isLenient()) {
parsePosition.index = oldStart;
parsePosition.index = oldStart;
return false;
}
}
break; // we succeed, we exit this loop

}
else {
if (isLenient()) {
parsePosition.index = oldStart;
parsePosition.errorIndex = oldStart;
return false;
}
break;
}

Using MiG Layout For Better Swing Development

Tue, 29 Apr 2008 18:27:00 +0000

I have forgotten a few libraries in my Better Swing Development article, and notably MiGLayout.

GridBagLayout is too verbose, and still feels too clumsy. This is why a while back I wrote a small tool to help visualize various GridBagLayouts for people who are not used to it. But it would have been much simpler to use a better layout instead.

MiGLayout is good, I managed to have good results without almost any practices on not so simple layouts. It also makes the code more concise.

_the_ Google 1998 paper

Fri, 25 Apr 2008 14:22:00 +0000

I have just read the anatomy of a search engine from S. Brin and L. Page. For those who don’t know, it is the Google paper. I have read other google labs papers in the past. What I like in this one is that you can follow how they came into having the Google ideas, how they assembled their ideas.

I should have read that a long time ago.

Fedora 9 Already Stable

Fri, 25 Apr 2008 11:33:00 +0000

Fedora 9

I “upgraded” my home computer to Fedora 9. “upgraded” because I reinstalled the OS instead of using the upgrade procedure. I have had so many issues with “partial” upgrades in the past (with any distro).

Although it is the preview/RC1 version, Fedora 9 is already as stable as a release IMHO. No issues so far, it feels more polished than Fedora 8. OpenJDK 1.6 is there. Firefox 3 is there. Not a single problem with kernel 2.6.25 (while I had plenty with the 2.6.24 ones).

SwiXml Review

Thu, 24 Apr 2008 12:19:00 +0000

Externalizing the screen layout is a natural step for many applications. It can be because a customer might want a slightly different presentation, or just for the sake of clearly separating layout code, or for using a kind of screen generator. Back in 2003, I wrote a small utility to help with GridBagLayout, nothing fantastic, but there was the feature of saving a generated layout in XML (or Beanshell) and using it in an application.

SwiXml saves a bit more than a layout as it instantiates the components as well, allowing to represent a hierarchy of components in XML. But it does not more than that. The "no generation" approach is interesting, I bet the author thinks the XML should be kept concise enough to be used easily without using a kind of generator. the XML is almost a mirror of Java code, but fortunately makes layouts more compact and easier to read. I tried version 1.51, and while it is quite well done, its documentation is a bit lacking and to me an important functionality is missing:

There is no "component" tag by default. One needs to explicitely name the class of component we want to add. I think there could be a component tag that instantiate an 0 parameter constructor by default, using reflection for example. The class of component to instantiate would be found through the field name.

Fortunately SwiXml is quite well done and this is not that big of an issue since you can just interpret new tags with 1 line of code:

engine.getTaglib().registerTag("datecombobox", DateComboBox.class);
for a Jide DateComboBox.

I said documentation was lacking, because in my first non Hello World trial, I wanted to use the standard constant names for GridBagConstraints (RELATIVE,REMAINDER, FIRST_LINE, LAST_LINE, ...). Putting <gridbagconstraint anchor="FIRST_LINE">did not work. By reading the code, I found out you could achieve it but you have to do <gridbagconstraint anchor="GridBagConstraint.FIRST_LINE">.

In the end SwiXml left me with a good impression. It is non intrusive, and handle well enough what it is done for. One major drawback, you can only use either BorderLayout or GridBagLayout or FormLayout (from JGoodies).

Better Java Swing Development

Thu, 17 Apr 2008 20:55:00 +0000

It has been a while since I have looked at the different options for building GUIs in Java. I like the approach taken by Eclipse, they provide not only GUI components, but a complete GUI application framework. Several years ago, I have stolen ideas from it (how they manage plugins) and applied it for Swing applications. This worked out quite well. The Netbeans platform now offers a more interesting alternative if you have to do Swing. The major drawback when compared to Eclipse, is that since the beginning Eclipse was all about making a platform, while Netbeans copied that idea only relatively recently and is less active about making it so. IBM uses Eclipse foundations for many of their products successfully.

Unfortunately on many projects, a Swing application is already there and all you have to do is add more screens, more forms. What I am trying to see here, is what could help Swing development is those cases.

JGoodies

JGoodies is old now, but can be quite useful in creating nice looking forms quickly. It is basically all around a layout better tailored to forms that the can do everything GridBagLayout.
It can help but it does not make a significant difference vs good practices with the GridBagLayout.

Jide Software

Jidesoft provides much more than JGoodies, but is not free (except the Jide Common Layer). I have used it in the past and found their API quite clean and easy to use. They offer more elaborated Swing components that should be part of the JDK. They started to go to the platform area only recently with their Jide Desktop Application Framework, which is not free either. I am not sure I would base an application around an unfree framework.

SwiXAT

It is a recent BSD licensed project based on top of Swix. It is basically MVC for Swix, with command binding in XML. I need to look more into it, it seems to provide an "application framework" as well where you have to launch your app through a SwixAt class (which IMHO sucks). I hope it is not a requirement and that only parts of it can be used and integrated into an existing application. I will hopefully have time to try it more concretely and will blog about it. In any cases it looks interesting.

SwiXml

An Apache licensed project, it seems like a decently used way of laying out swing components in XML, inspired by XUL. It might just be enough. Or is it just too similar to Java code translated to XML? I need to investigate this more.

JavaFX

JavaFX is probably the serious alternative to SwiXml. I like the idea of a better grammar to lay out and assemble components instead of using XML. Unfortunately it seems a bit invasive to start using for a part only of an application

I took a quick look at Swing Application Framework (JDNC2 redone once again). While it has interesting use of annotations, it seems very far from what Eclipse offers. And I don't feel like it will improve working with Swing a lot. I think except JavaFx, Sun seriously fucked around during many years with Swing. IBM has been much more focused and effective.

Fedora 8 vs. Ubuntu 8

Thu, 28 Feb 2008 14:23:00 +0000

I had the bad idea of trying (K)ubuntu 8.04 on my home computer. It worked for a few days, but as Ubuntu 8 is still in alpha, changes tend to break everything easily. At one point Wine stopped working, then the DVD, then the sound. I had crashes with USB plug/unplug. It was time to go back to a stable distro. I went back to Fedora 8.

I found out that I had missed some positive sides of Fedora 8 before. It compares quite well with the future Ubuntu 8. Both include pulseaudio. Pulseaudio in Fedora 8 is very well integrated, works well by default. I had to manually tweak things with Ubuntu, probably because it is an alpha.

Windows Vista Is Crap!

Fri, 08 Feb 2008 17:59:00 +0000

It has been already 1 year since Windows Vista is out. In my new company, I had a new computer with Vista on it. I quickly installed Ubuntu in dual boot. I was happy with Ubuntu for my work, until I received many Excel files OpenOffice was not able to read properly. I decided to give a try to Vista. My first impression of it when I had the computer was quite negative. I was not able to find things that used to be easy to find. It seemed slow and not very interesting.

The 2008 Java Web Framework

Tue, 22 Jan 2008 11:50:00 +0000

One of my first web framework was a custom one, based on the book Web Development with Java Server Pages. That was in 1999, when Struts was not yet known. But it was web-MVC, very near from what Struts gave later, without the custom tag stuff (custom tags were not yet part of JSP specifications).

Until 2007, Struts was IMHO the most used web framework and the default choice, even though many contenders appeared, with more features, better code. 2007 saw the maturity of some of these framework. As similar framework to Struts, I particularly like Stripes, because the API is very lean. Some component frameworks got a lot of buzz as well, Tapestry some time ago, and today Wicket. I tried both, and while they are very interesting, and can be very productive for some tasks (web application more than web site), I am not convinced they are the tech of 2008. Web Applications are better done with Flex, or GWT.

The major news in the past few years has been Ruby On Rails. Everybody tried to replicate it in every language (with their own twist): Django in Python, Lift in Scala, ErlyWeb in Erlang, Cake in PHP, Grails (along with now abandonned Sails, and less populair Trails) in Java. And some people fought for making the point they had the idea before Rails. But Rails is what made Rails-like development popular.
After trying Rails soon after it was out (way before 1.0) and more recently (just before 2.0), I was amazed by the progress of Rails. I was impressed in 2004 about Rails, because the approach was new and effective. I disliked a bit the choice of Ruby. In the end in the project we ended up rewriting the persistence layer because our application was very specific. And I ported the app very quickly to Java. In 2007, I was even more impressed, and did not need to dwelve into arcanes of Rails anymore, but merely using it was enough to do lots and lots of things.

In 2008 the Ruby On Rails mentality will continue to prevail. In the Java world, Grails is the most likely to benefit from it. But I would not be surprised if there was another new Rails-like contender in the game (Rails on JRuby?). It could also be something based around Spring as their current MVC solution is not very good and very old fashioned.

KDE4 Still Not Ready

Fri, 18 Jan 2008 19:08:00 +0000

I tried KDE4.0 last week, when it was everywhere in the news. I was very disappointed.

I have a long experience of trying beta software, not well known software. I have used Linux since 1998 (partly). I have even used windows beta at one point. And I have seen nothing as crappy as KDE4.0 release.

It is not usable at all.

Now I can understand what are KDE motivations to do such a bad release. It is to gain momentum in their project. KDE4.0 is in development at least since 2002 - 6 years! By creating a .0 release now even if it is an alpha, they have at least something officially out and hope to have developers starting to improve it. It is especially important for them because Gnome has, in the mean time, made regularly lots of improvements.

Fedora 8 - The Comeback

Sun, 23 Dec 2007 02:28:00 +0000

My previous post about Ubuntu Gutsy vs Fedora 8 was weak. It is difficult to find a very good distro. Depending on the computer, I have had different experiences. On some computers, Ubuntu really shines and work with minimal tweaking. On some others, Ubuntu is unstable/does not handle wireless correctly and Fedora is much more stable.

The main issues I can see with Fedora 8 are:

LVM by default. I don’t think it is a good idea to go LVM by default since lots of basic tools are still not handling it properly. And if you want to read your disk by something else than a distro with LVM you are screwed. Ext3 straight is imho a much wiser choice. Plus it is rarely a problem to resize partitions as it is not something one does often.
Fewer programs in the repositories available. Under ubuntu, I was using gtkguitune to tune my guitar, it was in the default repositories. It does not exist for Fedora and I did not manage to compile it due to too old dependencies (GTK 1.2). I found accordeur on sourceforge which is a better program and is available as RPM, so in the end I found something. But while searching I saw the choice was not as wide as with Ubuntu.
Packages too small: it is not exactly clear what packages you need to start compiling programs with Fedora. If you look in the default categorization, way too many things are silly to enable by default. Also generally package management seems less stable/less easy to use than with Ubuntu.

What I like:

Haskell Fibonacci Revisited

Wed, 12 Dec 2007 17:08:00 +0000

Recently, there was an interesting post about Haskell performance and Haskell parallelization showing Haskell could outperform C on a simple Fibonacci example.

A friend of mine, Peter (that I seem to manage to constantly piss off) thought about it on another level, saying you could achieve a _MILLION_ times better using a direct formula in C or Java, the Binet formula.

I decided to try as the improvement scale seemed a bit surprising. I first compared a Java recursive fibonacci with a Haskell one. Here are the results for Haskell GHC 6.6.1 vs Java 1.6.0 on Linux for fib(44):

Then I decided to check out the time for fib(44) or any fib at all, I was unable to measure precisely enough since it always came out as 0ms, in Haskell, or in Java. Looping out 10 million times, Java gave out 7.3s and Haskell something similar (but my method to loop 10 million times in Haskell is probably very bad).

The original post actually points to a link that describes various algorithms for Fibonacci. They basically say that for large n, the rounding is not precise enough, they also propose algorithms in log(n). I tried and was really impressed by the performance of those algorithms. Again I could not measure the difference for a single calculation between it and the binet formula as elapsed time is always 0. The binet formula becomes inexact already at n=71 in Java with doubles.

Of course the original post is still quite interesting, it shows how easy it can be to parallelize calculations in Haskell. But the example is silly as another algorithm can lead to 10 millions times the performance. Still Haskell performs well with the shit or good algorithm when compared to Java.

Ubuntu 7.10 vs Fedora Core 8 - Gutsy vs Werewolf

Tue, 20 Nov 2007 16:23:00 +0000

I was pretty happy with Ubuntu 7.10, but when Fedora 8 came out I decided to give it a try. Last time I tried it it was Core 2 or something like that, and it was NOT good.

At first Fedora 8 looks quite good, has a good Live CD install, reminiscent of Ubuntu. The positive side is that it is based on the latest Kernel. It manages my Thinkpad T42 very well (suspend, hibernate work). But after a few days, one notice Fedora is not as stable as Ubuntu, for example:

Apache DbUtils Completely Useless

Fri, 02 Nov 2007 17:17:00 +0000

I am disappointed about the Jarkarta Commons DbUtils project. I give a link to it, because it’s a bad project (even if written in clean code). It is very simple, but it really does not do much for you.

I was looking for a very simple abstraction of JDBC. I thought bringing Spring in my project would be overkill. After trying DbUtils, I think again. It does not help. It does not handle frequent cases well, and it does not save many lines of code.

I fell in the trap of Boolean.getBoolean()

Fri, 12 Oct 2007 16:10:00 +0000

I was struggling to find a bug in a very simple application, it ended up being something as simple as using the damned Boolean.getBoolean("true") call instead of Boolean.valueOf("true").booleanValue() call.

The Boolean.getBoolean method is something you almost never need to use, as it checks if a particular system property is true or false. There is a similar method for Integer.getInteger, and a quick google search shows I am not the only one to think those method should never have been part of the basic API for Boolean/Integer. It is too easy to confuse with parseBoolean/parseInt, especially as parseBoolean does not exist in JDKs prior to JDK 1.5 (parseInt is older).

I can not imagine the improductivity this method has produced given its part of one of the most used class in the world.

Fast Web Development With Scala

Tue, 25 Sep 2007 14:02:00 +0000

I am currently experimenting with Scala. It seems quite convenient for web applications. Using Tomcat, it is possible to have a very productive developer environment.
Here is a sample Embedded Tomcat you can start in a Scala project:

import java.io._;
import org.apache.catalina._;
import org.apache.catalina.startup._;

object TomcatScalaServer {
  
  val CATALINAHOME : File = new File("../newsbeef.com");
  val WEBAPPS : File = new File(CATALINAHOME,"webapps");
  val ROOT : File = new File(CATALINAHOME,"web");
  val HOSTNAME : String = "localhost";
  val PORT : int = 8080;
  
  def await() {
     while( true ) {
          try {
            System.out.println("sleeping 100s");
              Thread.sleep( 100000 );
          } catch {
            case ie : InterruptedException =>;
          }
      }
  }
  
  def start() {
    val server = new Embedded();
    server.setCatalinaHome(CATALINAHOME.getAbsolutePath());

    val engine = server.createEngine();
    engine.setDefaultHost(HOSTNAME);

    val host = server.createHost(HOSTNAME, WEBAPPS.getAbsolutePath());
    engine.addChild(host);

    val context = server.createContext("", ROOT.getAbsolutePath());
     context.setParentClassLoader(Thread.currentThread().getContextClassLoader());
     context.setReloadable(true);
    host.addChild(context);

    server.addEngine(engine);

    val http = server.createConnector(HOSTNAME, PORT, false);
    server.addConnector(http);

    server.start();
  }
  
  def main(args: Array[String]) {
    start();
    await();
  }

}

Here is a sample Scala Servlet outputing html directly. This is a simple example, but it shows something important. With Scala, the view layer can just be regular scala classes. There is no need for JSP or other templating languages as Scala already embbeds XML very nicely. By using the reloadable feature of Tomcat (there are also other pure Scala ways) and Eclipse autocompile, changes are instantanously taken in account.

import javax.servlet.http._;

class ScalaServlet extends HttpServlet {

  override def init() {
  }
  
  override def doGet(request : HttpServletRequest , response : HttpServletResponse) = {
    service(request, response)
  }
  
  override def service(req: HttpServletRequest,resp: HttpServletResponse) { 
    val pw = resp.getWriter();
    var output = <html>
    <head><title>Scala Servlet Test</title></head>
    <body>
      <h1>Hello World!</h1>
    </body>
    </html>
    pw.println(output);
    pw.flush();
  }
}

Now I am eagerly waiting for improvements in the Eclipse Scala plugin (Organise imports, class navigation).

2 Months of Ubuntu on Mac Mini

Mon, 27 Aug 2007 23:48:00 +0000

I am finally happy with my OS. I had previously [some complaints]({{ relref . “1-year-of-mac-mini–the-deception-point.md” }}) about MacOs X and the Mac Mini. It is now over, with Ubuntu, I am very happy of my quiet system.

I use Quod Libet for Audio, it has similar interface as iTunes, with more features (ability to play most audio formats). I chose Quod Libet instead of the standard Rhythmbox because of its practical mp3 tags handling. This also means that unlike iTunes, when I reimport my full library with another player, or on another computer, I have it all organized the right way, because the right meta data is in the audio files and not in a xml file that sometimes gets corrupted.

Spring Web Services, Finally!

Thu, 23 Aug 2007 11:22:00 +0000

Spring Web Services seems to be the technology I have been looking for recently. I am not a Spring bigot (too XML oriented), but here the Spring folks have something right.

I used to work with Web Services the simple way: create a java class (or EJB), expose it as Web Service through Axis or RAD, generating the WSDL in the process. And then a client would just be the reverse, take the WSDL, use a tool (Axis or RAD) that creates client Java classes from it automatically. Simple, easy.

Original Pattern: ServletRequest in ThreadLocal

Thu, 02 Aug 2007 12:07:00 +0000

After seeing Scala had elements of Erlang through Actors, I decided to take a closer look at the language. There is an interesting new web framework in Scala, called Lift. One drawback of Lift is that it seems to be very cutting edge and not that easy to grasp. While reading its source code, I stumbled upon a strange pattern: Storing the ServletRequest in a ThreadLocal.

I had not seen that before, and was wondering why one would do such a thing. It seems to be unintuitive. I found my answer through… GWT widgets. In this page, the author explain motivations behind doing such a thing:

Vim setup

Fri, 27 Jul 2007 16:13:00 +0000

Here is my Vim setup information for reference

in .vimrc or _vimrc, add at the beginning:

set langmenu=en_US.ISO_8859-1
set gfn=Bitstream_Vera_Sans_Mono:h9:cANSI
colorscheme oceandeep

First line is to avoid menus in French.
The font (you can find here) is simply the best programmer's font.
oceandeep mode can be found here .

Why Eclipse Is Better

Fri, 27 Jul 2007 14:37:00 +0000

Initially I adopted Eclipse instead of Emacs because it was more powerful to search code, and it allowed refactoring. I regularly tried other IDEs but always end up back to Eclipse, even though there has been less big improvements in Eclipse in the past years (but lots of small ones).

I just saw today that Eclipse allowed programmatic refactoring. Now that’s something quite amazing, and I don’t think other IDEs do that yet. Someone even had fun writing an Eclipse extension in Scala to add a particular kind of refactoring to Eclipse.

Tapestry5 vs Wicket: 1 - 0

Wed, 11 Jul 2007 12:49:00 +0000

Getting started with Tapestry 5 is easier than with Wicket 1.3. Some readers will complain that it is again the view of someone who has no deep knowledge of either Tapestry or Wicket. But I think it is important for projects to be easily accessible to developers. Wicket seems to have more buzz around these days, and has a detailed wiki with plenty of useful information in it. But that's the problem I see with Wicket, it is not simple to do simple things, that is why there is so much information to do simple things in the Wicket wiki.

Granted my test was based on a specific case for component frameworks, I was not so much interested into statefulness, I wanted to display a bookmarkable "user page" with content coming from hibernate.This kind of behaviour is quite general in web applications, especially in web 2.0.

It was relatively easy to have the page working with Wicket, although I was disappointed at their hibernate integration. Hibernate integration in wicket means either using the full databinder project, or creating your own solution. I chose the later based on source code from databinder, but I actually rewrote everything in the end. I was disappointed that databinder, a specific Hibernate oriented framework did not really handle Hibernate sessions the simplest way possible. Tapestry5 got that right. To manage Hibernate sessions right, I had to dwelve into Wicket code as no documentation offers insight about inner workings of wicket. The code was too complex for my taste. In my short experience, I saw it seemed the developers are changing it to the better, removing some unnecessary abstractions.

In the end I got frustrated many times with Wicket, and did not manage to have a bookmarkable page the way I wanted. You can have a bookmarkable page, but after some action on the page, it would become unbookmarkable. Furthermore, the structure of the URL is not very flexible without yourself rewriting completely the bookmarkable page feature of Wicket.

With Tapestry5, I was at first worried about the small amount of documentation on the site, the use of maven in the tutorial. I was wrong, documentation proved to be exactly what I needed, and detailed enough. It is much easier to understand how Tapestry5 works after reading the doc than Wicket. Concepts in Tapestry5 are simpler and more powerful. Maven use is in the end not that big of a deal, I am still not as comfortable with it but I am productive enough that it is not an issue, much more productive than with Wicket. The standard tutorial setup is a very good one.

Doing a bookmarkable page was trivial, it also was easy to have the format i wanted, and it was kept after action in the location bar. Hibernate integration was trivial, since Tapestry5 provides the tapestry-hibernate module, a few classes that helps managing the session and transactions for you. The only drawback is maybe the yet another inversion control system to learn. Tapestry5 IoC is very near from Guice in its philosophy. I wish Guice was made the default for IoC in Tapestry5.

To conclude, there is no doubt about it, Tapestry5 is the winner.

NetBeans 6.0M10 out without announcement yet!

Sat, 30 Jun 2007 07:15:00 +0000

I just found it while browsing netbeans website, here is the link. Netbeans is starting to be much more interesting that it used to be before 5.5, even though shortcuts are a pain, because so different from most other editors, and not always defined for important tasks. I like the all integrated feeling without plugin and slugishness by default.

Use ORM For Better Performance

Tue, 12 Jun 2007 12:59:00 +0000

This is not something I would have though a few years ago. It is something I learnt after working on many different projects, some using an ORM layer like Hibernate, Entity EJBs, or JDO, some using JDBC approach via Spring Templates or custom frameworks. Many projects that use ORM have performance problems, that don't seem that common with projects using JDBC. But the size of the database model of ORM projects is often much bigger than the one of JDBC projects (which actually makes sense). If you have only a few queries to do, why bother with ORM? This would be complexity for nothing.

But for most enterprise projects, the size of the database model is quite big, and the model itself can be complex (many relations between many tables). With this kind of model, ORM is more efficient. It is faster to develop with, creates less bugs due to string misspelled, or types badly read. It is also better performing. Doing 1 giant query to retrieve everything in 1 step is not faster, especially if you don't always need all the information retrieved. In a complex model, many cases are specifics, only useful in 10% of the cases. The temptation is high with a JDBC approach to do one giant query, because it is substantially longer (and more work) to do N queries. With ORM, it is a bit the opposite, by default N queries is easier to do. The problem is that N(ORM) tends to be very high if one is not careful with the mapping to avoid the N+1 problem. However it is simpler to reduce the number of queries by joining tables, rather than splitting queries, ORM performance optimization feels more natural.

Martin Fowler tends to be also pro ORM in its "Domain Logic and SQL" article. He also mentions something interesting about SQL query optimization:

It's also worth pointing out that this example is one that plays to a database's strengths. Many queries don't have the strong elements of selection and aggregation that this one does, and won't show such a performance change. In addition multi-user scenarios often cause surprising changes to the way queries behave, so real profiling has to be done under a realistic multi-user load. You may find that locking issues outweigh anything you can get by faster individual queries.

People Using Spring, EJBs Don't Know Basic JDBC

Wed, 30 May 2007 12:03:00 +0000

I recently found a bug in software we are developing. I traced it and found the root was improper JDBC handling. The application is written using EJBs, Spring and plenty of other relatively complex technologies. I was surprised that developers who were able to use all those technologies had no understanding of basic JDBC.

They fetched all the data (including double, decimal numbers) from the database as String using rs.getString() !

Wizards Bad For Productivity

Wed, 16 May 2007 14:56:00 +0000

IBM RAD comes with many wizards, to create EJBs, to create Web Services, do struts mapping… They are quite well done, making EJB < 3.0 usable, and Web Services look simple.

But wizards sucks at:

typos correction
repetition

But when you do a typo in your wizards, then all the files generated/changed are wrong, and you don’t necessarily know if you can just do a search and replace. Plus you don’t necessarily know all the files that were affected by the typo.

Less Productive With Maven2.

Sun, 29 Apr 2007 17:10:00 +0000

My first trials of Maven were failures. As I am stubborn, I tried again, on a new project, a quite simple one. It works, but it makes some easy things overkill. And the default way of using it makes a developer lose lots of time.

If I have a project with common classes, a standalone app, and a web app, then logically you do 3 projects, 2 of them depending on the common one. That’s how the default maven setup works, and that’s what their documentation presents. Now when using maven eclipse, this will create 3 project, none depending on each other. If you modify something in the common code, it won’t be seen by any of the other code, you have to publish it with maven first, this takes way too much time. Furthermore I did not see any way to force rebuild the common automatically from one of the other project. If you modify code in common and web app project, you need to call maven twice. I find all this very counterproductive, because you do those steps extremely often. Now there are probably some ways to do that with Maven2, but it is not the default behavior. I could add project dependencies in eclipse manually, and forget about maven while working in eclipse, but then the maven eclipse plugin is really useless. And you’ll face the same issues when you want to use maven tomcat deploy.

How to Build Good Software? Good network connection

Fri, 27 Apr 2007 12:40:00 +0000

Not having good internet connection can be problematic to download new libraries, read or search for documentation on development subjects. But not having a good internal network connection is killer of productivity. It means sometimes not being able to access integration, preprod or even production environment, or ssh session not responding in the middle of an action. As software makes an increasing use of the network, it means not being able to test or to use correctly all kind of software.

How to Build Good Software? Private office, again

Wed, 25 Apr 2007 15:01:00 +0000

Apparently it’s more a habit of French companies to have big open spaces with no separation at all between people. There is nothing more annoying than having people in conference call in front of you while you are trying to work on something completely different. French people forgot the cubicle part in the American open space idea. So sometimes the room is just a big mess, everybody being able to disturb you anytime. Even if I have no private office, please give me at least a cubicle.

How to Build Good Software? Welcome newcomers

Wed, 18 Apr 2007 20:17:00 +0000

Some companies do it naturally, some really don’t. In small companies, it is almost natural, people will make a newcomer productive very quickly. In a big company it’s not the same game.

Some important points are:

Computer ready the first day, well sized (right ram, right power, developers are not MS office users), right OS. I had experience with having not the right amount of ram, not the right version of OS, not the right user rights to install and use critical software for my work, and all those were known from the team. I also hate when companies give the cheapest computer available for developers/architects. On top of that badly configured computers take often a month to be ready in big companies. It just does not makes sense.
Network access. I have seen people coming for a short contract and not having network account or email account before 1 week.
Give documents to read, show applications the person will work with. Involve newcomer in new decisions on his project.
Some big companies have it right. I remember my internship at IBM Germany, where on the first day I had a box waiting for me with computer inside, that I had to unpack and install (with OS/2) for my use. I think this is the best way for developers/tech people. And then they recommended excellent reading on the subject I would be working on. It’s not that difficult.

Project Estimations And Fibonacci Sequence.

Fri, 13 Apr 2007 13:01:00 +0000

I was recently in a meeting where use case complexity was estimated using numbers in the Fibonacci sequence. I was surprised by the choice of the Fibonacci sequence. Why not any sequence? Why a particular one? I googled and found the culprit, Mr Mike Cohn in his book Agile Estimating and Planning. It’s actually not a bad sequence to choose, since the scale is increasing constantly, so by picking up numbers in this sequence, you can quite accurately describe estimation. If you have defined complexities of 1,2,3,16,17 corresponding to 5 different use cases then obviously 16 or 17 denotes the same complexity, and it would be surprising that you can really distinguish both. You need an ever increasing scale. But a power of 2 scale might not be precise enough (steps are growing too fast).

How to Build Good Software? Use a bug management software, really.

Fri, 13 Apr 2007 10:56:00 +0000

This will seem obvious, unfortunately, when people are involved, nothing is that obvious. It’s not because you setup a bug/feature management software that people will use it. You have to force people to go through the bug management software each time they want something fixed. If you don’t do that some people will keep sending incomplete mails, or worse call you to get something fixed, that will be forgotten in a week. It is also very useful to avoid receiving 10x the same request from the same person.

How to Build Good Software? Have a good build process

Thu, 12 Apr 2007 09:54:00 +0000

Important points are: standard code hierarchy, automatic download of dependencies, a distribution command with versioning support and source control interaction, a simple command to build each part of the project. In Java best candidates are a sophiticated ant build, or maven2. Maven2 is quite good since it forces you to do some of those steps, even though I think ant can’t really be avoided for many more specific tasks.

Once, I got wrong code before I left for a customer site, because code given to the customer was not checked into a source repository everytime before it was given to the customer. Furthermore the code I had could not be built because of missing jars, and the project structure forced some source directories containing source code common with other projects to be in a very specific directories at a particular level, because project isolation was poor. It took me 7 days to understand their very awkward build process.

How to Build Good Software? Private Office

Wed, 11 Apr 2007 10:52:00 +0000

In an open space, people keep on coming to discuss various issues with various people, issues that have nothing to do with your work. You end up either being distracted, or annoyed by the increased noise level.

Apparently at Microsoft, they have private offices for each programmer. It might be extreme, paradoxally not in the XP (extreme programming) sense, but it is much better than open space for productivity. In XP, it’s almost the opposite with 2 developers working often together.

How to Build Good Software? Lay Off Quickly.

Tue, 10 Apr 2007 11:12:00 +0000

If you have to lay off in your job, do it quickly. I don’t understand companies that want to keep someone as long as legally possible when this someone wants to leave. First the employee won’t be as motivated, but more importantly, you will continue to train that person to your company’s software and ways of work. This would be much better used on another person, that will stay in the company. Time for layoffs should exclusively be used for knowledge transfer.

How to Build Good Software? Talk to people, especially the ones you don't know well.

Fri, 06 Apr 2007 17:02:00 +0000

Someone modified a simple launch script on a integration machine. This pissed off the author of the script. Why?

Just because the guy who modified the script never worked before with the author of the script. If only the author had been notified verbally or by mail of the modification, he would have been happy. Furthermore this would increase the quality of the change since the new guy might have made a change that has other impacts, that the author will best evaluate quickly.

Find Grep And Vi Keys Small Memo

Thu, 05 Apr 2007 18:48:00 +0000

I tend to forget this now and then to grep on a specific list of files:

find . -name "*.xml" | xargs grep "iwantthis"

And I also tend to forget the vi keys. Small extract:

h - move left one character
j - move down one line
k - move up
l - move right
$ - go to the end of the current line
0 - go to the beginning of the current line
G - go to the last line in the file
15G - go to line 15
control-F - forward one page

control-B - backwards one page

n (N) - next (previous) in search mode (/ or ? forward or backward)

s/OLD/NEW/g - replace on current line

%s/OLD/NEW/g - replace every occurence in file (or use 0,$ instead of %)

x - delete one character

Update I have to add the standard replace in multiple files via sed command. Here is an example of how to move your eclipse workspace to another directory:

find . -name "*.xml" | xargs sed -i "s,c:[/\\]java[/\\]eclipse,d:/eclipse302,gi"

How to Build Good Software? MS Press Code Complete says Measure twice, cut once

Thu, 05 Apr 2007 14:14:00 +0000

I am starting a series of posts about software construction, based on good and bad experiences while working for my many employers.

“Software construction (a.k.a. architecture), is the only activity that’s guaranteed to happen on every project”. It is not rare that projects or new features are abandoned either because they are too costly, or because they are not needed anymore, or because they are other more important priorities.

In the same vein: “Measure twice, cut once”. “Measure twice, cut once” is highly relevant to the construction part of software development, which can account for as much as 65 percent of the total project costs. The worst software projects end up doing construction two or three times or more. Doing the most expensive part of the project twice is as bad an idea in software as it is in any other line of work.

Java Concurrency In Practice Book Review

Wed, 14 Feb 2007 16:33:00 +0000

My reference book on Java concurrency is Doug Lea's Concurrent Programming in Java: Design Principles and Patterns. He is one of the authors of this new book, Java Concurrency In Practice. There is also Joshua Bloch, author of Effective Java, that many people love (but I am less a fan of it, even if I would recommend it to Java newbies), and author of Java Puzzlers, that I found more fun.

With such authors, I had relatively high expectations. I was surprised that there is not much material in common between Doug Lea's book and this one, which is a good thing. It's a different presentation, that focuses on different problems.

There is some very interesting material, I especially enjoyed chapter 16 on the Java Memory Model. There are not many books with information on it. I did not know that initializing a final member in the constructor was providing thread safety for the accessor method of the corresponding field if there is no other modifying method, no synchronization needed. And not having the member declared as final was breaking the safety.

I also had not heard of Amdahl's law before. It seems quite intuitive, but then the example in the book about processing parallel tasks using a synchronized LinkedList (improvement up to 3 threads, no more later due to time spent on synchronization) shows that the reality is not that intuitive.

I enjoyed how the book is written, and how information is presented. There is for example their concept of publication and escape (in chapter 3) which tells you important things, like not starting a Thread in a constructor (and if you still want to do it they present a nice way to do it).

I think this book is a simpler read that Doug Lea original book, and probably a better introduction (that still goes very deep) to Java concurrency programming.

Why are you not using Entity EJB 1.1?

Wed, 07 Feb 2007 16:38:00 +0000

I am currently quite busy learning about Websphere Commerce. I was surprised to notice that they still use Entity EJBs 1.1. My experience was that many companies turned away from Entity EJBs, sometimes from the beginning, preferring TopLink or Hibernate (later).

I know that Entity EJBs are more heavy to use than Hibernate, but with the proper tooling support, it’s not really a big issue. Are there more fundamental reasons? There are probably some mapping limitations, but IBM implementation is apparently quite good in term of features.

Switching to Windows

Thu, 25 Jan 2007 01:04:00 +0000

That’s it, I have done it, the unthinkable. I have switched from MacOs X to Windows XP. My idea was first to install Ubuntu on the Mac Mini, but if you want to keep the Mac Os X install, the methods I have read on the web required to use bootcamp. At around the same time I was annoyed by problems I had with bluetooth and Skype in MacOs.

So, with bootcamp installed I decided to give a try to Windows. And it works quite well on the Mac. As I use it mostly as a media box (movies, photos, jukebox, browser and tablature reader) and not really as a developer, I find Windows quite convenient. I have always preferred Picasa to iPhoto. And moving back to Picasa for my photos is a joy. I can’t believe the speed difference between the two apps, and it’s not a matter of memory, I have 2gb. The crappy ways iPhotos deals with my folders is over. I am also happy to use again foobar2000 instead of bloated (and limited (only MP3s and MP4s)) iTunes. In my quest to look for a convenient setup, I discovered ProjectM visualisation, also available for MacOs X (with iTunes) and Linux. I am quite impressed with the animations. I have the impression that graphics are much faster under Windows than under MacOs, it’s probably due to much better optimized drivers.

Social (Web) Revelation: Tags Are Lame

Tue, 16 Jan 2007 10:26:00 +0000

I had a revelation yesterday while thinking about del.icio.us, the most well known social bookmarks site. I find out that I almost never use my tags, and I am often not satisfied by results when I search using tags. What use can you make of 100s of tags? in the end only less than 10 are usable to classify very different stuff. And even then compared to a search it’s not a much useful classification.

1 year of mac mini - the deception point

Mon, 08 Jan 2007 15:56:00 +0000

I am less enthusiastic about the Apple experience than I was one year ago.

I am quite enthusiastic with the hardware in general, except the keyboard. The computer is small and quiet, and there are not many competitors at this price on the market.

Experience as a developer

I feel frustrated with MacOs X as a development environment, when compared with Ubuntu Linux. I always miss the multiple desktop, despite expose magnificient look. The standard behaviour for the terminal, bash, vim, etc. are more convenient in Linux. Installing software and updates is much more convenient on Linux thanks to the open source repositories. The key bindings for special characters is awful for a programmer on the Apple keyboard, because non standard, and not even written on the keyboard. Installing linux on a mac mini intel is not trivial, but I am considering it seriously.

Tired Of Bad Singletons

Thu, 04 Jan 2007 11:50:00 +0000

While looking through some code for a project, I saw that:

public static final ProductYP instance = new ProductYP();

public ProductYP()
{
    if (instance != null)
      throw new RuntimeException("Only one instance allowed");

    prods = new HashMap();
}

public static ProductYP getInstance()
{
    return instance;
}

And I don't think it was done by a newbie... It's actually not far from being correct, it's just that the guy obviously does not know about private constructors. I have seen several broken singleton implementations in previous projects and had several debates on the double-checked locking pattern (it works since JDK1.5 with volatiles but is useless). I am upset to see another half broken implementation. Everybody should have read at least this.

What have you seen as broken singletons?

Declaring Your Logger - No Problem

Fri, 29 Dec 2006 11:45:00 +0000

I used to like the java specialists newsletter, most news used to be a bit challenging. Nowadays however the quality is lower. In the latest news, the author proposes to use the StackTrace to get the class name, in order to declare a Logger independently of any explicit reference to the class name.

While this is a clever hack, it still requires some code to be duplicated in every class, compared to Aspect/IoC approach. But was there really a problem with declaring loggers the usual way in the first place?

Very Quick Django Trial -> Bye Bye Django

Thu, 21 Dec 2006 17:48:00 +0000

I tried a bit django for one very simple thing, where django is not of much use, but it was a way to try it with something else than examples on the site.

Even if django templates features seem very good and very convenient when reading the docs (for example the extends keyword), I found out that the template system is very annoying. Firstly one has to learn another templating system. Secondly, and much more importantly, the debug facilities for templates are close to non existent. I have a better memory using ruby on rails templates (which are not the best either).

Java templates, based on Velocity, JSPs or JSF are much nicer to debug. PHP (which I consider an ugly language that can unfortunately often gets the job done) used to be crap at debugging, but with eclipse plugin phpeclipse it is much better these days.

I think templates play a very important role in web application development, and I am surprised that a framework with so many good ideas fails in reality (at least for me) because debugging templates is a pain. Hopefully it will become better one day, but until then bye bye Django.

A Proof That JavaScript Sucks.

Fri, 15 Dec 2006 16:51:00 +0000

Google developed GWT.

Why would GWT be so well acclaimed if JavaScript was a good language. When you talk about GWT to someone (a developer preferably), the first reaction is often

great I don’t need to do any JavaScript

I recently discovered another similar open source project, ZK. The first thing you can read on the website is

ZK is an open source Ajax Web framework that enables rich user interface for web applications with no JavaScript and little programming.

Building With Ant + Maven Antlib?

Thu, 14 Dec 2006 10:51:00 +0000

I discovered a new baby in the build world. Some open source projects use a combination of ant and maven. I thought you could vaguely use ant from maven. But I did not know of Maven Antlib that allows you to use maven2 from Ant. Why would you use Maven2 from Ant?

because Maven2 requires to adapt your project to its standard hierarchy, or to have excellent Maven2 skills to create your own archetype for each existing project.
because everybody knows Ant, but Ant does not manage dependencies well. Maven2 Antlib can be used just for that: managing dependencies.

The Hibernate Network Proxy, A Dumb Idea.

Mon, 04 Dec 2006 14:41:00 +0000

A friend of mine had a good idea for a Java project named “hibernate proxy”. I renamed it “hibernate network proxy” because hibernate proxies are already a well known concept among hibernate users (it allows lazy loading through a Java Proxy class).

The hibernate network proxy, is a way to use hibernate remotely, that is to proxy hibernate calls through TCP (or even HTTP) to another machine, that will execute the session.saveOrUpdate or the session.load, etc. calls and return the corresponding objects, serialize them and deserialize them for use on the client.

Workaround When Eclipse Fucks Off With CVS

Tue, 28 Nov 2006 11:42:00 +0000

Until recently, I always found Eclipse CVS support to be excellent. The synchronize view was really a big plus to do clean commits, without forgetting any file.

But I have encountered several problem with Eclipse 3.2+:

On one project, it keeps putting the build directory into CVS, and I can't find how to change that setting. Each time I open the project I have to change the CVS/Entries file.

On another, where I have lots of files in, Eclipse delete some of the files (checked in CVS) regularly, although not often. It won't retrieve them back again from CVS. It seems to be the same files each time. I have no clue why.

The diff sometimes tells me the whole file has changed while there is absolutely no change.

I could not make the merge work. It kept on complaining about some parameters.

And now the workaround for all those cases:

The good old CVS command line.

I never remember how do to a merge with the command line, so I each time go to google and read this merge example page. Here are the very simple steps if you are in a branch and want to merge:

cvs update -A mypoem.txt (use head as local repository)

cvs update -j release-1 (merge release-1 changes into local repository)

```
cvs update mypoem.txt
```
```
cvs commit mypoem.txt
```

Maven: Game Over, Try Again

Fri, 24 Nov 2006 17:51:00 +0000

I am writing only as a newbie on Maven2. I tried to use it several times on small projects, but each time I found out that the time involved to make it work with an existing project was too much a pain in the a**. When confronted to very simple hierarchies that are not the standard of the documentation, I was a bit lost on how to arrange things, except changing the whole hierarchy to adapt to Maven.

How JDepend Changed My Java Packaging

Fri, 13 Oct 2006 09:27:00 +0000

One important feature of Java language is the package keyword. It helps a lot in modularizing your code. But how exactly one should use it is not that clear.

I have been on projects where 100s of classes are in a same package and more often where you have packages for every 2 classes. Often, packages are chosen so as to split functionalities. But often as well, people are packages maniacs and create way too many of them, because they want to sort things out, not necessarily applying a consistent logic. For example, I am sure many of you have seen the “blahblah.exceptions” packages where you dump exceptions classes.

You Know IOException? Think Again!

Thu, 14 Sep 2006 10:59:00 +0000

I was amazed today to find out that there was no constructor IOException(String, Throwable) or IOException(Throwable) in JDK1.4 and JDK1.5. It is finally in JDK1.6, I can’t believe it took Sun that much time to change that.

So the workaround is:

`IOException ioe = new IOException("message");`
`ioe.initCause(e);`
`throw ioe;`

It can also be written as:

`throw (IOException) new IOException("message").initCause(e);`

It is not a major problem, but still. We can all thank the guy who reported that as a bug to Sun in 2004.

Good Software Books - 2006 version

Mon, 11 Sep 2006 14:27:00 +0000

Here is an update of the most interesting software books I found interesting. I already made such a list in 2005:

Object Oriented Software Construction, 2nd Ed, by Bertrand Meyer. This made me understand why OOP is important, what is important in OOP and why it is still relevant. It contains lots of important guidelines you can apply to better design programs. One famous quote is "Real systems have no top". Code Complete (Microsoft Press) covers some of the same ideas as Meyer's book, and some more pragmatic issues that arise in software projects ("measure twice, cut once"). I also liked the chapter on "table driven design".
Concurrent Programming in Java, Second Edition, by Doug Lea: there is all you need to know about programming in a multithreaded environment in it. The chapter on synchronization should be a must read for every Java developer.
Design Patterns by the GoF: simply the best presentation of most common design patterns. I have a glance on it once in a while.
Artificial Intelligence through Prolog, by Neil C. Rowe (Prentice-Hall): if you don't remember much about Prolog, it's a good book. It details how Prolog interpreters work.
File Systems Forensic Analysis, by Brian Carrier (Addison Wesley): everything you need to know about disks and file systems, every trick to recover lost data. To go more in depth into file systems theory, you can read the free Practical File System Design with the Be File System from D. Giampaolo, the creator of BeFS.
Mind Hacks (O'Reilly): you probably already bought that one. Not about software, but excellent.
Practical Issues in Database Management by Fabian Pascal (Addison Wesley): short but concise book on relational database theory. F Pascal is of CJ Date school of thoughts, "null are evil". You don't need to read it if you have read any other similar book (by CJ Date for example).
Inside The Java Virtual Machine, by Bill Venners (McGraw-Hill): there are other books on the same subject. While this book has some bad reviews, I found it an easy read, and it explain well enough for me all the inner details of Java. I found "Programming for the Java Virtual Machine" not better, and with some bad example of a Prolog language for the JVM (1 full chapter for this to present a way too simple thing to be of any use).
Programming Jabber, by O'Reilly: I have read it 2 years ago, I enjoyed how they made you go through building a Jabber server.
Lucene In Action: the only book about Lucene, some of the info is useful to understand how it is designed.
Hibernate In Action: if you have to use hibernate, this is the book to read. It presents different ways of using it.
some Javascript+DHTML(+CSS) book: useful if you have none. They often have good tricks to solve the usual problems. In the AJAX days, it becomes useful again. I find O'Reilly's "Javascript - The Definitive Guide" good to have an API reference (as ebook), and "Javascript and DHTML Cookbook" has the most useful recipes.
Lean Software Development by Poppendieck (Addison Wesley): easy to read, good presentation of software dev management problems and pragmatic solutions. My favorite on the subject. Another good one but very specific is "Requirements Management" (MS Press).
UML Distilled by M Fowler: always useful to have if you need to draw UML diagrams.

I Get Spring

Wed, 23 Aug 2006 18:04:00 +0000

When you google up Java Spring, one of the best results is a post from crazybob called “I Don’t Get Spring”. For a long time, I shared a similar opinion. But now that I have used it, I get it. I only use it for defining replaceable services, so when I talk about Spring, I mean spring-core and spring-beans.

These two packages are not big, and have only very few dependencies. So it is quite easy to use Spring in any project, be it small or not. It is actually easier than using NanoContainer because Spring has less external dependencies.

Java Serialization vs .NET Serialization - Java Perverse?

Tue, 22 Aug 2006 12:55:00 +0000

Did you know what happens in Java when you serialize a subclass of a non serializable class? I was surprised by the answer: it works!

Unfortunately it is not a good thing, because it will serialize fields from your subclass and no fields from the parent class. So you'll end up with a half serialized instance.

In .NET, it breaks at runtime, throwing an exception, which is I think, much more logical, because then you don't end up with half data somewhere.

Java Code:

import java.io.ByteArrayInputStream;
       import java.io.ByteArrayOutputStream;
       import java.io.ObjectInputStream;
       import java.io.ObjectOutputStream;
       import java.io.Serializable;
       
       
       
       public class Test
       {
         public static class Toto 
         {
           public String me;    
         }
         
         public static class Toto2 extends Toto implements Serializable
         {
           public String you;
           public String toString()
           {
             return me+" "+you;
           }
         }
         
         public static void main(String[] args) throws Exception
         {
           Toto2 t = new Toto2();
           t.me = "it's me";
           t.you = "it's you";
           System.out.println("t="+t);
           ByteArrayOutputStream b = new ByteArrayOutputStream();
           ObjectOutputStream oos = new ObjectOutputStream(b);
           oos.writeObject(t);
           ObjectInputStream ois = new ObjectInputStream(new ByteArrayInputStream(b.toByteArray()));
           System.out.println("u="+ois.readObject());
         }
       }

will output:
t=it's me it's you
u=null it's you

.NET Code:

using System;
using System.Collections.Generic;
using System.Text;
using System.IO;
using System.Xml;
using System.Xml.Serialization;
using System.Runtime.Serialization.Formatters.Binary;

namespace ConsoleApplication
{
    public class Toto
    {
        public string me;
        public override string ToString()
       {
            return me;
       }
    }

    [Serializable]
    public class Toto2 : Toto
    {
        public string you;
        public override string ToString()
       {
            return you + " " + me;
        }
    }

    class Program
    {
        static void Main(string[] args)
        {
            Toto2 t = new Toto2();
            t.me = "it's me";
            t.you = "it's you";
            using (FileStream fs = File.Create(@"c:\test.bin"))
           {
                BinaryFormatter bFormatter = new BinaryFormatter();
                bFormatter.Serialize(fs, t);
            }
            Console.WriteLine("t=" + t.ToString());
            Toto2u = null;
            using (FileStream fs = File.Open(@"c:\test.bin", FileMode.Open))
            {
                BinaryFormatter bFormatter = new BinaryFormatter();
                u = (Toto2)bFormatter.Deserialize(fs);
            }
            Console.WriteLine("u="+u.ToString());
            Console.ReadKey();
        }
    }
}

will throw an exception.

5 Minutes of Google Spreadsheets

Thu, 17 Aug 2006 12:10:00 +0000

Today I noticed a Google Spreadsheets link in my gmail screen. I had read about it but never bothered to try before. In the finance industry, a lot of traders use excel, so I was wondering if Google spreadsheets could be another fit.

Unfortunately for Google, under Linux at least, I don’t find Google Spreadsheets usable for anything else than storing and sharing some information, not often updated. Although I admire the engineers that managed to write the Javascript behind Google Spreasheet, it is way too slow for using it in interesting ways. Editing is slow, copy/paste is slow, sorting is slow.

Procedural Programming in an OO language

Tue, 08 Aug 2006 12:35:00 +0000

OO is an old buzzword, that is not required anymore to get an employment. Recruiter seems to prefer SOA, Web Services, and in France, “mutualisation” and “urbanisation”. Sometimes I really wonder if OO made it.

I am sure many of you are confronted with programmers that love procedural programming in many of your projects. They might use an OO language but in the end will organize everything by “type”, split invariably state and logic. Everything will be so much better stateless. And we will create lookup maps to glue everything back together. In a way I feel they remove the OO of the language.

Back To Real Java, Bye Bye J2EE

Wed, 26 Jul 2006 16:26:00 +0000

I changed job recently. In this new job, it is refreshing to see Java used like in the old days, without the J2EE layers, and without the extra IBM layers of my previous job. Granted, the fresh Java approach does not apply to many projects, because a lot of apps are just about interfacing a database with a web interface. But Spring success showed that even for many of those projects, fresh Java approach with a small framework is enough.

Moving weekly Javablog stats to a new blog

Mon, 26 Jun 2006 17:25:00 +0000

Javablogs.com top 10 weekly/monthly/yearly entries were starting to pollute my blog too much for my taste. It is more appropriate to dedicate a blog to them. That is what I should have done in the first place as it is very easy to have many blogs with blogger.com.

So you’ll find at javabuzz.blogspot.com the weekly top 10 most read entries on Javablogs, and sometimes more.

JSF Was Too Hard for Experienced Developers

Fri, 23 Jun 2006 14:59:00 +0000

While starting to look into Seam, I noticed that all examples use JSF for the view, and there is no alternative to it. If someone like Gavin (from Hibernate fame) thinks JSF is usable, then I probably overlooked something when I looked into it a few years ago, when JSF was the craze of the moment.

At the beginning, JSF looks very similar to ASP.NET. But I have a small experience with ASP.NET, and ASP.NET is quite simple to understand and use. You can throw up inexperienced developers at it, they will manage to create something that works quite quickly. ASP.NET feels quite natural once you understand the postback thingy. JSF is another beast. Maybe part of it is due to the fact that the Java world has no excellent tools integration like Microsoft has (IBM RAD, one of the most advanced concerning integration, is quite far off). But there are also many technological reasons; when I read that article about JSF shortcomings with JSP, I was shocked that even to do very simple things, you would screw up, because simple things can be very complicated in JSF.

On EJB Restrictions - Can You Open a Socket?

Mon, 19 Jun 2006 11:48:00 +0000

During job interviews, one of the frequent questions is

“What restrictions are imposed on EJBs?”.

I had a chat recently with someone about it, and we still don’t agree on the answer.

He believes we can’t open sockets, while I believe it is perfectly allowed, if you don’t retain the socket connection between method calls. His arguments are based on an analogy with files. After all in Linux, files are sockets. Here are his interpretations on a document about EJB restrictions on java.sun.com:

Non Blocking with Traditional Java IO - On the Use of InputStream.available() and Thread.sleep()

Thu, 08 Jun 2006 16:31:00 +0000

Some time ago I did quite a lot of IO in Java and I yet did not see this way of reading a InputStream from a Socket:

    InputStream in=channel.getInputStream();

    channel.connect();

    byte[] tmp=new byte[1024];
    while(true){
      while(in.available()>0){
        int i=in.read(tmp, 0, 1024);
        if(i<0)break;
        System.out.print(new String(tmp, 0, i));
      }
      if(channel.isClosed()){
        System.out.println("exit-status: "+channel.getExitStatus());
        break;
      }
      try{Thread.sleep(1000);}catch(Exception ee){}
    }
    channel.disconnect();

This comes from a piece of code from an example of JSch , a good ssh client in java. A work collegue had the bad idea to remove the Thread.sleep call and was struggling with why it would randomly work.The way I would have done it is the following:

    InputStream in=channel.getInputStream();

    channel.connect();

    byte[] tmp=new byte[1024];
    int bytesRead = 0;
    while((bytesRead = in.read(tmp,0,1024) >= 0){
      System.out.print(new String(tmp, 0, bytesRead));
    }

    if(channel.isClosed()){
      System.out.println("exit-status: "+channel.getExitStatus());
      break;
    }
    channel.disconnect();

This has the advantage of being more readable and having less secret spices in it. In the first code, the call to available() is non blocking, meaning that without the Thread.sleep(), there will never be the time for the socket buffer to fill up. But is the first code more efficient or the second code?

I did a search on google to understand the interest of the first code. The only advantages I found in the first code are the possibility to interrupt the thread running the code and a finer grained control on timeouts.

There is a lengthy explanation by Doug Lea in his book "Concurrent Programming in Java". This book usually provides excellent explanations, and is a must read for anybody doing concurrent programming. But this time, about this subject, I did not find him that clear.

There is a more simple explanation in a course from San Diego State University (see last example)

A read() on an inputstream or reader blocks. Once a thread calls read() it will not respond to interrupt() (or much else) until the read is completed. This is a problem when a read could take a long time: reading from a socket or the keyboard. If the input is not forth coming, the read() could block forever.

As usual, you should not rely on all what you read on the web, as this page (SCJP Questions & Answers) testifies:

Q. When will a Thread I/O blocked?

A:

When a thread executes a read() call on an InputStream, if no byte is available. The calling Thread blocks, in other words, stops executing until a byte is available or the Thread is interrupted.

Still I am wondering if the second code would not just go into IOException (socket timeout), on timeout (adjustable with Socket.setTimeout ) and release the Thread then. Do you have an idea when the first code could be better?

Is Java Flawed - a big advantage of Python/Ruby/(your favorite interpreted language)

Fri, 02 Jun 2006 17:33:00 +0000

Java is supposed to be much better to build big projects, because of static type checking, and all the rigour around the language. But how many of you have seen medium sized projects taking more than 30 minutes to build.

At work, they have a standard J2EE project, with only about 50 EJBs, hundreds of JDO classes, and standard classes. Between the JDO generation, EJB generation, EJB dependencies calculations, and packaging, it takes 20 minutes. And the project is not doing that much. One can optimize to avoid dependencies calculations and it would then take about 12 minutes. But still.

Using Linux to Recover Fucked Up Windows Data

Thu, 01 Jun 2006 01:52:00 +0000

Recently, one of my relatives computer under Windows XP, refused to boot. There was no way of fixing it with Windows Install CD as partition table seemed corrupt to Windows. I tried everything in an 2003 Ultimate Boot CD, but nothing worked out.

Someone gave me an install cd of Ubuntu Linux, and it managed to read the data. Well sometimes only. The erratic behaviour was due to a bad ATA cable. This probably was the cause of the corruption in the first place. Anyway with a new cable, Windows was still not able to read its data. But Ubuntu Linux, now working well, was able to, without having anything to configure (except mounting the drive).

Java HTML Parsing Example With htmlparser

Tue, 30 May 2006 11:02:00 +0000

Every week, I post javablogs top 10 most read blog entries on this blog. The reason for it was that I don't look at what's happening on the week-end and this will pickup interesting stories from the weekend, and I also don't watch javablogs everyday. Overall I find it quite good to be uptodate with interesting stuff happening on javablogs.

As mentionned in an earlier post my library of choice to do the parsing is htmlparser (on sourceforge) because it's free, open source and because I am lazy and did not want to do my own. If you know a better open source library, feel free to add a comment about it, I'll be glad to hear about it. htmlparser is not the easiest library to use, there are many entry points and it's not immediately clear which one to choose. So I post here how I used it if it can save a few minutes to people having to do this task.

  private static Entry parseEntry(String content) throws ParserException
  {
    final Entry entry = new Entry();

    final NodeVisitor linkVisitor = new NodeVisitor() {
      
      @Override
      public void visitTag(Tag tag) {
        String name = tag.getTagName();

        if ("a".equalsIgnoreCase(name))
            {
              String hrefValue = tag.getAttribute("href");
              if (hrefValue != null && !hrefValue.startsWith("http://"))
              {
                if (!hrefValue.startsWith("/")) hrefValue = "/"+hrefValue;
                hrefValue = "http://javablogs.com"+hrefValue;
                //System.out.println("test, value="+hrefValue);
              }
              if (hrefValue != null)
              {
                hrefValue = hrefValue.replaceAll("&", "&amp;");
                tag.setAttribute("href", hrefValue);                
              }
            }
      }
    
    };
    
    NodeVisitor visitor = new NodeVisitor() {

      @Override
      public void visitTag(Tag tag) {        
        String name = tag.getTagName();
            if ("span".equalsIgnoreCase(name) || "div".equalsIgnoreCase(name))
            {              
              String classValue = tag.getAttribute("class");
//                LOGGER.debug("visittag name="+name+" class="+classValue+"children="+tag.getChildren().toHtml());
              if ("blogentrydetails".equals(classValue))
              {
                Pattern countPattern = Pattern.compile("Reads:\\s*([0-9]*)");
                Matcher matcher = countPattern.matcher(tag.getChildren().toHtml());
                if (matcher.find())
                {
                  String countStr = matcher.group(1);
                  entry.count = new Integer(countStr).intValue();
                }
                
              }
              else if ("blogentrysummary".equals(classValue))
              {
                try
                {
                  tag.getChildren().visitAllNodesWith(linkVisitor);
                }
                catch (ParserException pe)
                {
                  LOGGER.error(pe,pe);
                }
                entry.description = tag.getChildren().toHtml();                 
                entry.description = entry.description.replaceAll("\\s+", " ");
              }
              else if ("blogentrytitle".equals(classValue))
              {
                try
                {
                  tag.getChildren().visitAllNodesWith(linkVisitor);
                }
                catch (ParserException pe)
                {
                  LOGGER.error(pe,pe);
                }
                entry.title =tag.getChildren().toHtml(); 
                entry.title = entry.title.replaceAll("\\s+", " ");
              }              
            }
            
      }

    };
    Parser parser = new Parser(new Lexer(new Page(content,"UTF-8")));
    parser.visitAllNodesWith(visitor);
        if (entry.title != null)
        {
          return entry;
        }
        else return null;
  }

First Steps With EhCache

Thu, 04 May 2006 18:10:00 +0000

If you need to cache objects in your system, Ehcache is a simple cache written in Java, widely used and well tested. I will present here a short tutorial on how to use EhCache for people who don't want to look around the documentation at first, but just want to test if it works in their project and to see how easy it is to setup.

Installation

Download Ehcache from the Download link on http://ehcache.sourceforge.net. Current release is 1.2.

Unpack Ehcache with an unpacker that knows the tgz format. For unix users, it is trivial, for windows users, 7zip is a free (and open-source) unpacker. It is probably the most popular, but there are other ones like tugzip or izarc or winrar.

In your java project you need to have ehcache-1.2.jar, commons-collections-2.1.1.jar and commons-logging-1.0.4.jar (versions numbers may vary) in your classpath, those libraries are shipped with ehcache.

Cache Configuration

Write an ehcache.xml file where you describe what cache you want to use. There can be several files per project, several cache descriptions per file. I use here a persistent cache. Configuration file is well described at http://ehcache.sourceforge.net/documentation/configuration.html

</ehcache>

Null vs. Errors

Wed, 26 Apr 2006 16:16:00 +0000

I am not particularly a fan of JCS (Jakarta Cache System) as I find ehcache code very clean and simple. But I have to say the author has some good comments on the site:

Nulls vs. Errors

I started to support ObjectNotFoundExceptions for failed gets but the overhead and cumbersome coding needed to surround a simple get method is ridiculous. Instead the JCS return null.

For having seen too many times the ObjectNotFoundException "pattern", I can only agree!

Algorithms in Java (Third Edition) Book Review

Wed, 26 Apr 2006 12:26:00 +0000

The book Algorithms in Java is huge, but unlike the usual huge books, the content is very interesting. It can be used as reference material, or as toilet book (to learn things while you're wasting time in the toilets).

You will learn simple things, like what is the "raison d'être" of linked lists. The author gives very good examples to illustrate his propositions. He explains through the sieve of erathostene and through Josephus problem the advantages of arrays or linked list.

You will learn step by step everything that is to be known in algorithms. Recursion, divide and conquer, Tree knowledge will be useful for the later sorting and searching chapters.

The chapter on Hashing will make you understand very clearly why the source of String.hashCode() is

public int hashCode() {

  int h = hash;

  if (h == 0) {

     int off = offset;

     char val[] = value;

     int len = count;

     for (int i = 0; i <  len; i++) h = 31*h + val[off ++];

     hash = h;

  return h;

There might be too much info on different types of sorting algorithms and the book becomes there more a reference book than anything else. But overall, you will learn plenty with this book. It is very well written, complete, and will refresh one's memory. I find it useful to read back things I learnt after a few years as I then have a very different view of the subject, and I pay closer attention to some details I completely missed the first time (sometimes).
Tags: java book review

Caching HTTP Responses in Java

Wed, 19 Apr 2006 18:14:00 +0000

Caching HTTP response can dramatically improve performance of your app if what you generate is in reality not very dynamic. There are many free caching frameworks in Java. Most popular seem to be ehcache, oscache, jcs and JBoss Cache.

ehcache is quite simple to use and its code is clean. They have a CachingFilter that you can put in your webapp server to cache transparently HTTP responses. However as the framework only allows you to store Objects (which makes sense for most uses), I was wondering how they cached the HttpResponse which is a stream. I was a bit disappointed by the answer, they just create a copy ByteArrayOutputStream and call toBytes() to store it in the cache. While this is optimal for a memory cache store (the whole response will anyway be in the cache, although I am not sure if they check for particularly big responses to avoid caching those or try to cache those) I don’t think it is that good for a disk cache store.

Javablogs.com 2005 Top 20

Tue, 18 Apr 2006 16:42:00 +0000

How To Use Java With Blogger: A Tutorial

Tue, 18 Apr 2006 16:36:00 +0000

Blogger has a REST API. I use it to retrieve particular posts or to post transformed data. There is no Java API that I know of, but you will see here it is not very difficult to interface with Blogger API in Java using plain old XML.

Using libraries commons-httpclient and DOM4J it would be quite easy to implement your own Java Blogger API as the following code will suggest.

Authenticate
All requests need to be authenticated and are done in HTTPS. I use common-httpclient to perform requests. Here is how to setup the client:

private  HttpClient initHttpClient()
{
  HttpClient client  = new HttpClient();
  List authPrefs =  new ArrayList(2);
  authPrefs.add(AuthPolicy.DIGEST );
  authPrefs.add(AuthPolicy.BASIC);
  client.getParams().setParameter (AuthPolicy.AUTH_SCHEME_PRIORITY, authPrefs);
  client.getParams().setAuthenticationPreemptive(true);
  Credentials defaultcreds = new UsernamePasswordCredentials(user, password); 
  client.getState().setCredentials(new AuthScope( "www.blogger.com", 443, AuthScope.ANY_REALM), defaultcreds);
  return client;          
}

Get Your Posts
To retrieve the posts, you just have to query the right url, and parse the XML response. I prefer to use DOM4J, because of its handy asXML() method to print a node as XML. For simplicity I use a Map to store an XML entry.

public Collection getPosts() throws  HttpException, IOException, ParserConfigurationException, SAXException, DocumentException 
{
  GetMethod get = new  GetMethod("https://www.blogger.com/atom" +"/"+blogId);
  int  statusCode = client.executeMethod(get);
  if   (statusCode !=  HttpStatus.SC_OK)
  {
    throw new RuntimeException(" Could not make HTTP  request properly: " +get.getStatusLine());
  }
  InputStream response  = get.getResponseBodyAsStream();
  SAXReader reader = new  SAXReader();
  Document doc = reader.read(response);
  Collection posts = new ArrayList();
  List entries  = doc.getRootElement().elements("entry");
  if (LOG.isDebugEnabled()) 
  {
    LOG.debug("found  "+entries.size()+"  entries");
  }
  for (int  i = 0; i <entries.size();i ++)
  {
    Element entry  = (Element) entries.get(i); 
    Map m = new  HashMap();
    for (Iterator it = entry.elementIterator();it. hasNext();)
    {
      Element detail =  (Element) it.next();
      String name = detail.getName();
      if (name.equals("link"))
      {
        m.put("link ",detail.attribute("href").getValue());
      }
      else if (name. equals("content"))
      {
        m.put("content",detail.asXML());
      }
      else
      { 
        m.put(name,detail.getTextTrim());
      } 
    }
    posts.add(m);                   
    if  (LOG.isDebugEnabled())
    {
      LOG.debug( "found="+m.get(" title")+", url= "+m.get("link")); 
    }
  }
  return  posts;
}

Create XML for a new Post
Nothing particular here, just XML production.

private String createXmlForCreatePost(String postTitle, String postContent) throws   IOException, DocumentException
{
  SAXReader xmlReader =  new SAXReader();
  xmlReader.setValidation(false );
  Document doc = DocumentHelper.createDocument();
  QName rootName  = DocumentHelper.createQName("entry", new  Namespace("", "http://purl.org/atom/ns# "));
  Element root = doc.addElement(rootName);
  Element title = root.addElement("title"); 
  title.addAttribute("mode"," escaped");
  title.addAttribute("type ","text/plain");
  title.setText (postTitle);
  Element generator = root.addElement("generator ");
  generator.addAttribute("url" , "http://31416.org");
  generator. setText("31416 Java Generator ");
  Element content = root.addElement("content ");
  content.addAttribute("type" , "application/xhtml+xml");
  //Element  div = content.addElement(DocumentHelper.createQName("div",new  Namespace("","http://www.w3.org/1999/xhtml")));
  //div.add(...); //YOUR XHTML HERE
  StringWriter result = new  StringWriter();
  XMLWriter writer = new XMLWriter (result);
  writer.write(doc);
  writer.close();
  return result.toString();
}

That's it

Last week Javablogs.com top 10

Tue, 18 Apr 2006 11:02:00 +0000

Web Services No Better Than CORBA?

Mon, 10 Apr 2006 15:23:00 +0000

Richard Monson-Haefel (RMH), author of the famous Enterprise Java Beans (O'Reilly) and of a Web Services book is making an alarming claim about JAX-WS, the new Java web services standard:

JAX-WS still sucks!

In an effort to write a client to real world web services (for Google, Amazon, Ebay), he mostly failed. Errors seem to be largely related to WSDL to Java conversion. Jason Green, a JBoss developer, managed to get quickly a web service working for Ebay, but analysis by RMH suggest that hundreds of classes were generated for this simple one method web service.

Recently I had to try to do CORBA client classes by hand in Java since the rmic tool was not working that well to generate classes that could be run on a client JVM of different version than the server JVM. It proved to be a no-go in the end, because of the complexity involved not only in coding, but more on maintaining those modified generated classes. This would have required coding a tool dedicated for that purpose. We chose to pass simpler objects, differently, which looks just like a hack to avoid having CORBA issues. I am amazed that after that many years of CORBA, there are still simple cases where it does not work properly automatically.

Seeing that there is the same kind of problem with Web Services (IDL to Java translation) makes me wonder about Web Services "improvement" over CORBA. And WSDL is definitely less readable than CORBA IDL.

Is your UML Composition French or American?

Fri, 07 Apr 2006 11:40:00 +0000

The French Way
In many French books/lessons (a lesson here, another from caen university , one from UQAM), UML composition does not seem to mean the same as their English counterparts. They write:

Si l'agrégat est détruit (ou copié), ses composants le sont aussi.

les composants (en tant qu'ils sont composants) ont une durée de vie incluse dans celle de leur composé.

translated to English:

If the aggregate is destroyed (or copied), its components are destroyed (or copied) as well.

components (while they are components) lifetime is included in the one of their aggregate

But then what does it mean to have a cardinality of 0 on the aggregate side?
Few French lessons give an answer (UQAM does, maybe because it's half French only). The component can be created before the aggregate but must be destroyed when the aggregate is.

The American Way
Most English books/lessons describe composition more accurately. Ootips definition is simple and clear:

Composition [...] is exactly like Aggregation except that the lifetime of the 'part' is controlled by the 'whole'. This control may be direct or transitive. That is, the 'whole' may take direct responsibility for creating or destroying the 'part', or it may accept an already created part, and later pass it on to some other whole that assumes responsibility for it.

However Martin Fowler in UML distilled explicitly writes:

The "no sharing" rule is the key to composition . Another assumption is that if you delete the polygon, it should automatically ensure that any owned Points also are deleted .

By using the word "assumption" he is avoiding a clear meaning. So I looked up in the UML specifications from uml.org. The specs are actually very near ootips definition and even more explicit:

An association may represent a composite aggregation (i.e., a whole/part relationship). Only binary associations can be aggregations. Composite aggregation is a strong form of aggregation that requires a part instance be included in at most one composite at a time. If a composite is deleted, all of its parts are normally deleted with it. Note that a part can (where allowed) be removed from a composite before the composite is deleted, and thus not be deleted as part of the composite. Compositions define transitive asymmetric relationships—their links form a directed, acyclic graph.

So it turns out that some French definitions were not wrong, but imprecise. Their true meaning becomes only clear after reading the English UML specs.

Ootips has also a good example of difference between aggregation and association. If you have an aggregation relation between two Node entities, it means you represent a tree of Nodes, not a graph of Nodes. See the link for more details.

Are UML Aggregation and Composition Good Concepts?

Tue, 04 Apr 2006 11:12:00 +0000

Yesterday a work collegue made the remark that the notions of aggregation and composition were not very clear. I was a bit surprised at first. But I actually almost always use just associations relations, sometimes composition, I was not sure anymore about the difference between aggregation and association and the one between composition and aggregation. I had to look up in a book to remember what it was all about.

The French book we looked at was really bad. In the end we did not understand fully the differences except a notion of "stronger coupling", some kind of "asymetry"!?! The difference between aggregation and composition was more explicit since in a composition, the "child" can only have one parent.

The UML Distilled book is much more explicit and I recommend that book to anybody doing UML:

Aggregation is strictly meaningless ; as a result, I recommend that you ignore it in your own diagrams .If you see it in other people's diagrams, you'll need to dig deeper to find out what they mean by it . Different authors and teams use it for very different purposes.
Composition is a good way of showing properties that own by value, properties to value objects (page 73), or properties that have a strong and somewhat exclusive ownership of particular other components.

Bertrand Meyer prefers to describe relationships using only two concepts, client and heir, basically UML association and specialization and therefore avoids the distinction between aggregation and composition and association:

B is a client of A if every object of type B may contain information about one or more objects of type A.
B is an heir of A if B denotes a specialized version of A.

He then uses "class invariants" to specify the nature of the client relationship.

SOA and B. Meyer

Fri, 31 Mar 2006 15:08:00 +0000

While rereading parts of Object Oriented Software Construction from B. Meyer, I still find valuable information I overlooked. In the part about functional decomposition where he argues pro and cons of top-down / bottom-up approaches, he has the intuition of the current SOA hype. By showing that

Real systems have no top.

He suggests that the only way to build complex software is through a service oriented architecture. Of course SOA is a very old concept. Meyer example of Operating Systems is a fine one.

Tired Of 'My 10 Best XXX' Blog Posts

Thu, 30 Mar 2006 10:32:00 +0000

It's incredible how many posts on popular aggregators are just about a list of stuff, and how except one word they sound the same.
Only today on Javaworld, I can read:
" Ten Things Every Java Developer Should Know About Unix"
"My 10 Favorite Scripting Languages "
"7 Must-Have Programs for Windows"

They all seem to have read the
"10 best ways to have your post popular on delicious"

It's a bit boring.

Bilinear Gaussian Lanczos? Downsampling!

Fri, 17 Mar 2006 17:32:00 +0000

Not easy to choose a downsampling algorithm. Some links:

mplayer: advises bilinear for quality and describes artifacts for each algo
a blogger: advises lanczos. This links shows pictures produced by each algo.
digital image interpolation: seems to have a preference for bicubic, what photoshop uses.
videohelp: tested various algorithms and think Bicubic spline is the best.

People don't seem to agree. If you want to check the theory:

Getting Started With DOM, XOM, DOM4J by Parsing an RSS Feed - An Experience Review

Fri, 17 Mar 2006 10:53:00 +0000

Recently, I looked for a way to get info from some particular blog entries of mine on blogger.com. Blogger used to offer an XML-RPC API . They even designed a version 2 of the XML-RPC API that does not seem to have ever been put in production. Or maybe I did not manage to make it work. I had no problem to make v1 work, however. I used apache XMLRPC v2, it was very simple to use. Unfortunately information accessible through XML-RPC Blogger API was incomplete for me. Furthermore, it is very likely that this API will disappear soon as it is deprecated since 2002.

Blogger wants you to use their Atom API. It is not XML RPC anymore, you have to do the parsing by hand.

The DOM Experience

I thought "no big deal, I will use DOM for it". I don't need performance and wanted a quick way to solve my problem, plus DOM does not require X number of libraries. It was easy to use regular DOM until I was frustrated by not being able to get the <content> element full text easily as it is sometimes XML. I did not want to hand code a method to do that as I thought it should be done by the XML library.

The XOM Experience

I heard previously of a simple XML parser, efficient, with an API that had been well designed, XOM. I looked at the API, there was a toXML () method to return the node content as XML (children included), sounded good. I saw there was even XPath support, and thought, great, it will simplify my code a bit. I will get the blog entries by just querying for " /feed/entry". No luck, it did not work, it returned 0 results. So I looked for mistakes in my code, did not find obvious ones. I tried other queries like " //feed/entry" or "//entry", same thing, not the right results. There must have been something wrong in my code, or maybe the XPath engine in XOM has particular settings to deal with RSS feeds (they contain various xmlns declarations). The point is that I got frustrated, it was supposed to be very simple, and in reality, not so!

The DOM4J Experience

I had experience with Dom4j before, just once, to build XML, not to parse it. I had relatively good memories of my Dom4j experience for that so I decided to try it out on my problem. At first I found Dom4j API a bit confusing as there are so many methods on most used classes. This is because Dom4j is DOM compatible. But I quickly understand the logic of it and found some very useful methods, namely Element.elements(name) to get all children elements by name. Of course, they have an asXML() method like XOM. There is also XPath support.
I tried the XPath on Blogger RSS without success again. There really must be a trick to get it to recognize RSS. But with the elements("entry") method, I very quickly got the same with not much more code, and it worked.

so DOM vs. XOM vs. DOM4J = 0 - 0 - 1

Example Code:


SAXReader reader = new SAXReader();
Document doc = reader.read(response);
Collection posts = new ArrayList();
List entries = doc.getRootElement().elements("entry");
if (LOG.isDebugEnabled()) 
{
        LOG.debug("found "+entries.size()+" entries");
}
for (int i = 0; i <entries.size();i++)
{
        Element entry = (Element) entries.get(i); 
        Map m = new HashMap();
        for (Iterator it = entry.elementIterator();it.hasNext();)
        {
                Element detail = (Element) it.next();
                String name = detail.getName();
                if (name.equals("link"))
                {
                        m.put("link",detail.attribute("href").getValue());
                }
                else if (name.equals("content"))
                {
                        m.put("content",detail.asXML());
                }
                else
                {
                        m.put(name,detail.getTextTrim());
                }
        }
        
        posts.add(m);                   
        if (LOG.isDebugEnabled())
        {
                LOG.debug("found="+m.get("title")+", url="+m.get("link"));
        }
}

Bad JDK Design Example

Thu, 16 Mar 2006 15:02:00 +0000

Have you ever used the Authenticator class? It is standard since JDK 1.2. This allows you to establish authenticated HTTP requests. OnJava has an in-depth article presenting it. Now here is my story about it.

The other day I just wanted to do that a simple authenticated HTTP request. You would expect it to be very simple to do in the modern internet world.

It is actually not very difficult to do, by hand, a JavaWorld article presents this way. Unfortunately it uses internal JDK sun classes for Digest and I am uncomfortable to use that since it is not guaranteed to work on any JVM. So I have to download jakarta commons-codec to just do a simple authenticated HTTP request. I really wonder why some of those codecs are not in the JDK (not as sun.* classes but as java.* classes).

Current Popular Subjects In Java Blogs

Wed, 15 Mar 2006 20:00:00 +0000

Based on my blogs entries and success of other blog entries, I present the current trendy subjects and the not trendy ones:

In:

Design Patterns
Ruby on Rails
EJB (bashing)
RSS
Spring
Hibernate
Javascript and AJAX
Java Virtual Machine

Out:

RMI
Portal
Jini

Neutral

JDK 5 features

As you can see most of the popular stuff is fairly common, not that new. I will do a finer, less common, analysis later if I have time.

Non Fiction Books (Mostly Software Programming) Before 2003

Wed, 15 Mar 2006 15:03:00 +0000

A page with useful reference for me as I don't own each one of these books. It can provides you a very short review non-commercial biased.

2003

Programming Jakarta Struts (O'Reilly - Chuck Cavaness) much better than Manning book. Explains struts very clearly with a more interesting example. Not enough on how to design your application for use with the evil ActionForm. Almost nothing on Struts 1.1 features. Nothing on common problems encountered while building web applications, or even about on how to design them using struts.
Struts In Action (Manning) not much more about struts than what's available on struts website. Their example throughout the book is the same login example. Struts 1.1 features, esp DynaForm are mentioned but not really presented. On the positive side, there is a very useful Validator reference at the end and a too short mention of different strategies for the ActionForm.
Contributing to Eclipse (Kent Beck - Erich Gamma) funny little drawings inside to make some good points.
The Design Patterns Java Companion(Adison Wesley - James W. Cooper) A must read for clean swing development. Essencial design patterns inside. Nice tip about doing deep cloning in java.
Java Threads (O'Reilly - Scott Oaks and Henry Wong) very clear presentation of multithreading programming in Java. Explain how to write your own thread scheduler.
J2EE expert one on one (Wrox) pragmatic view of J2EE. Many interesting comments like why avoiding stateful session bean if you are a little bit concerned with performance. Some strange benchmarks at the end of the book showing velocity templates being 10x faster than JSPs.
Java Distributed Computing(O'Reilly - Jim Farley) very good to learn more about RMI because it starts with a custom distributed object system.
Java Enterprise in a Nutshell (O' Reilly - David Flanagan, Jim Farley, William Crawford, and Kris Magnusson) good complementary book, provides a good basis. I liked how RMI use in EJBs was detailed, Servlet chapter is also excellent (but don't look for design tips/help in it - it is a book on the BASICS).
Enterprise Javabeans, second edition (O'Reilly - Richard Monson-Haefel) second lecture made me better appreciate that book, contains a lot of fundamental concepts to know. Very detailed on transaction management. However there is nothing I remember in it about EJB restrictions. In short incomplete but not bad.
Structure And Interpretation Of Computer Programs (MIT Press)

2002

How Mac Os X Made Me Discover a Bug in my Java Prog

Tue, 14 Mar 2006 20:15:00 +0000

Yesterday I had some free time to finally find out why MacOs X would not display my little applet properly. I checked various JDKs for MacOs, no difference. I checked if it was due to antialiasing use, no luck. I actually found out there was an error in the way I displayed images. I did not call repaint() between each image change. Strangely, it worked fine on Windows with many JDKs. Anyway now my Benham Disc Applet is working on Apple computers as well.

Del.icio.us Firefox Toolbar Update

Mon, 13 Mar 2006 19:45:00 +0000

I updated my little delicious toolbar that provides autosave feature for me. I did not bother reworking the new delicious extension as I don't use its new functionalities. However people are free to do so if they want with the code from this toolbar.

With Firefox 1.5, display of saved documents was not working properly anymore due to a change in Firefox handling of security permissions to view local files. It is fixed now.
Tags: delicious, toolbar, extension, firefox

Why did Netscape choose Javascript?!?

Thu, 09 Mar 2006 17:10:00 +0000

Last year, I have helped building an AJAX web application, where the web client was composed of only one web page and tons of javascript. The 1 web page for a site approach is similar to Gmail and many full AJAX sites. This approach looks quite elegant. Using JSON (and JSON templates) we were able to separated fully presentation logic and business logic and componentization seemed to occur naturally.

But I remember how often I complained on how bad Javascript was to code. I find it does not encourage good practices at all. The object orientation of Javascript is really a pain to use. If I had not a small experience with Python before, I would have thought scripting languages were really bad at building applications, even not so big ones. I find that even for good programmers, Javascript is a challenge, as it seems to always be a struggle to organize properly your code with it.

New Mac Mini

Tue, 07 Mar 2006 20:51:00 +0000

Here is my new Mac Mini Core Solo. I am quite happy with it since it is very quiet. It makes a good jukebox/movie/server machine. Apple was very quick in shipping it (2 days)

However I was a bit disappointed by iPhoto 6, it is not as nice as Picasa to sort out pictures (no IPTC support). I did not try Java on it yet. I just found out my little Benham circle applet was not working properly on it.

Features That Ruby Lacks and Java Has

Fri, 10 Feb 2006 15:57:00 +0000

Bertrand Meyer describes in his book " Object Oriented Software Construction (2nd Ed)" qualities that a good object oriented language should have. Java has almost all the features. Ruby is much farther off, which does not mean it is not a good language, I think it's very good to write some types of programs quickly (should I call them scripts?), syntax is quite nice, but it does lack the following useful features:

Assertions:
The language should make it possible to equip a class and its features with assertions (preconditions, postconditions and invariants), relying on tools to produce documentation out of these assertions and, optionally, monitor them at run time.
Information Hiding (Java is not good either on that one, the protected keyword is of dubious value):
It should be possible for the author of a class to specify that a feature is available to all clients, to no client, or to specified clients
Static Typing:
A well-defined type system should, by enforcing a number of type declaration and compatibility rules, guarantee the run-time type safety of the systems it accepts.

Genericity:
It should be possible to write classes with formal generic parameters representing arbitrary types.

More pragmatically, library choice and performance difference between Java and Ruby might be the most decisive.

Tags: book review

What Is 'Modern' Java Compilation?

Tue, 07 Feb 2006 11:12:00 +0000

Occasionally in Ant you can see messages like this:
"[javac] Using modern compiler"

What does this mean?

In Ant you have the property " build.compiler" to specify if you want to use a classic or modern compiler. Now what do they mean by modern or classic. Well, they call classic compiler the compilers of JDK 1.1 and 1.2 and they call modern compiler the ones of JDK 1.3+. They made that distinction because a classic compiler does not support the same options as modern compilers: the semantics of javac tool changed in JDK 1.3.

This terminology can easily be confused with java class file compatibility. Java class file compatibility is changed using the "-target" option of javac tool. One can specify in ant to compile with modern compilers and a target 1.4, the result is likely to not run on JVM 1.3. One can specify modern and a target 1.1, the resulting classes will run on JVM 1.1, but ant build.xml file will not be usable with JDK 1.1 without changing modern to classic.

I find Ant choice of word a bit confusing as using a "modern" compiler has little to do with the resulting class files.

RMI Nightmares - Non Interoperability

Tue, 31 Jan 2006 18:39:00 +0000

These days we tried to make Websphere and Jonas interoperate. Some EJBs are deployed in Jonas and we just try to call them from Websphere. The most natural way is to use RMI (with JRMP) for that. Jonas uses JDK1.4.2 (conforms to J2EE 1.4) and Websphere IBM JVM 1.3.1 (conforms to J2EE 1.3). Because of RMI serialization, an EJB from Jonas will not work on Websphere, because this EJB has references to J2EE 1.4 classes which are different from J2EE 1.3. The exact root cause is not exactly clear to me, it probably has to do with the Naming Factory and the EJB Stub, but we tried every possible thing here without success.
Update: JRMP worked between JBoss 3.X on JDK 1.4 and Websphere 5.X (IBM JDK 1.3) using JBoss NamingFactory. So RMI interoperation seems quite unpredictable. One thing is that you have to pay attention to the naming factory you use. If it is a special library like in JBoss case, make sure it is compatible with your JDK.

One solution is to use RMI-IIOP, as it does not rely on serialization. It almost works. In our case we use Websphere Portal that needs to have security enabled. You might ask what security has to do with all that. Well, enabling security on websphere means that CORBA exchange will also be secured (using CSIv2). Tuning Jonas to understand what websphere asks is something we did not manage to do (and we don't know if it is possible at all).
Update: We did manage to make it work by disabling Jonas CSIv2 support.

So there we are using Axis to do the communication!
Update: Axis is not needed anymore as RMI-IIOP works as advertised.

There is an interesting benchmark of Axis vs JRMP vs RMI-IIOP (pdf). Axis is more than 10x slower than pure JRMP. Application servers use of JRMP seems to matter a lot, JBoss looks bad.

What Is Clustering About – An answer to Spring is not designed for scalability

Mon, 16 Jan 2006 16:57:00 +0000

I wanted to reply to Diego Parilla post about scalability, but my reply ended up being too long not to post it over here. I don’t know if I am the only one here but I don’t fully understand his post and arguments.

Clustering the web server and clustering the model are just two parts of clustering, not exclusive.

In a LAMP application, clustering the model is traditionally done by clustering the DB (MySQL) which is a no brainer. Clustering the web server does not always implies sharing the “HTTP session”. For example you can limit yourself to use cookies. For many web apps this works well.

My Dream Job

Thu, 12 Jan 2006 11:42:00 +0000

In my dream job, I would have freedom to play with and build software based on new ideas I have.
In my dream job, I would work with creative people, luminaries.
In my dream job, I would have a good enough salary.
In my dream job, I would not have to worry about the company stability.
In my dream job, I would have freedom in my work schedule.

Get paid for doing what you really want (i.e. your ideas) is possible. But without worrying about the next 6 months, I have not seen that.

Ogg better than MPC and MP3 - AAC crap

Wed, 04 Jan 2006 11:47:00 +0000

The latest well done (e.g. blind) listening test comparing Ogg, MPC, MP3, AAC at about 180kbps shows that Ogg is superior to all other compressing formats. It’s not that easy to find relevant tests. Many people claim to be able to notice huge differences between CD and MP3 and yet, but confronted to a blind listening test, they are very surprised how difficult it actually is. Hydrogenaudio has a good community of audiophiles.

What's Your Favorite Design Pattern

Tue, 27 Dec 2005 12:18:00 +0000

Once in a job interview, someone asked me

What’s your favorite design pattern?

I was first surprised at the question. Usually a design pattern is about solving a problem not about a personal preference. I wondered if it was a trick question of some sorts. I was hesitating with my answer and asked more about the meaning of the question. I just had to really give the pattern I liked the most! I really should not have but I ended up pleasing my interviewer and said “the decorator pattern”. At least its name is somehow related to arts.

Professional Portal Development Book Review

Tue, 27 Dec 2005 12:08:00 +0000

I just finished reading Professional Portal Development With Open Source Tools (Wiley).

The Good:

I enjoyed Chapter 1, it gives a good presentation of Portlets for someone who does not know much about it. It is a chapter one can use as a reference to verify some specific details of how Portlets are working. It is a bit similar to JSR 168 specifications, but I found it more detailed and well written.

IBM Websphere Portal Strategy

Wed, 07 Dec 2005 17:48:00 +0000

I digged a bit deeper in a current project where they use IBM Websphere Portal Server (WPS) with IBM Web Content Management (WCM). I found out that both products collide and disintegrate.

WCM is like a regular content management system, quite flexible and performant on its own. It has a nice way to separate layout concern from content concern. It has a flexible cache. It has a user repository to set different permissions. Its architecture is even so well done (very modular) that you can tailor it to your needs to do things a content management system is not supposed to do. It could do most of what you would do with a Portal system. And that’s the problem.

Portlets in an AJAX World

Tue, 29 Nov 2005 16:29:00 +0000

Recently, I have had my first encounter with Portlets although Portlets are not very new. IBM had their own Portlet API before the JSR 168 was final in 2003. Unfortunately both approaches are quite different and IBM does not recommend the use of their own API.

Portlets were a needed concept for a long time. I worked in 2000 on a java portal system and it would have been handy at that time. Portals are everywhere since the internet bubble.

But Portlets do not do that much either. Sites designed with strict and intelligent rules can bypass the portlets need for example by using a combination of custom tags and java beans.

What shocks me a bit more is that while not doing much, it looks a bit old fashioned already. With AJAX you could build very flexible and user-friendly Portlets replacement. This AJAX-let is actually probably what Microsoft has already done on their start.com web site. Now with AJAX, I just wish Javascript would evolve (a lot) or die to be more OO friendly.

Good Software Books

Thu, 17 Nov 2005 21:24:00 +0000

A friend asked me for a list, here is it. I found those books the most interesting ones among those I have read about software in the past year or two:

Artificial Intelligence through Prolog, by Neil C. Rowe (Prentice-Hall): if you don't remember much about Prolog, it's a good book. It details how Prolog interpreters work.
File Systems Forensic Analysis, by Brian Carrier (Addison Wesley): everything you need to know about disks and file systems, every trick to recover lost data. To go more in depth into file systems theory, you can read the free Practical File System Design with the Be File System from D. Giampaolo, the creator of BeFS.
Mind Hacks (O'Reilly): you probably already bought that one. Not about software, but excellent.
Practical Issues in Database Management by Fabian Pascal (Addison Wesley): short but concise book on relational database theory. F Pascal is of CJ Date school of thoughts, "null are evil". You don't need to read it if you have read any other similar book (by CJ Date for example).
Inside The Java Virtual Machine, by Bill Venners (McGraw-Hill): there are other books on the same subject. While this book has some bad reviews, I found it an easy read, and it explain well enough for me all the inner details of Java. I found "Programming for the Java Virtual Machine" not better, and with some bad example of a Prolog language for the JVM (1 full chapter for this to present a way too simple thing to be of any use).
Programming Jabber, by O'Reilly: I have read it 2 years ago, I enjoyed how they made you go through building a Jabber server.
Lucene In Action: the only book about Lucene, some of the info is useful to understand how it is designed.
Hibernate In Action: if you have to use hibernate, this is the book to read. It presents different ways of using it.
some Javascript+DHTML(+CSS) book: useful if you have none. They often have good tricks to solve the usual problems. In the AJAX days, it becomes useful again. I find O'Reilly's "Javascript - The Definitive Guide" good to have an API reference (as ebook), and "Javascript and DHTML Cookbook" has the most useful recipes.
Lean Software Development by Poppendieck (Addison Wesley): easy to read, good presentation of software dev management problems and pragmatic solutions. My favorite on the subject. Another good one but very specific is "Requirements Management" (MS Press).
UML Distilled by M Fowler: always useful to have if you need to draw UML diagrams.

Good luck!

Apple Rumors and X86 first move

Wed, 16 Nov 2005 17:59:00 +0000

I find the mac mini a particularly compelling computer. It is small, noise friendly, relatively cheap. If you want to have a cheap home server (based on Linux or MacOS X), this is quite a good choice as you probably don’t want to hear all the fans of bigger computers the whole day. It’s also a very good candidate for a media box: you can connect a huge hard drive for mp3s, and watch your DVDs, photos on a nice LCD (and MacOS helps here).

7 Years Later: Still EJBs

Mon, 07 Nov 2005 16:29:00 +0000

In 1998, a student of my school had a project with Sun about EJBs. This was the beginning of EJBs, they were not public, and there was no J2EE. At that time he found it utterly complicated, and had difficulties to understand why they made it so complicated.

In 2000, I worked for a company who had built in-house most components of a today J2EE application server. For example, they had their own unified messaging system, on the JSP side, they were already using custom tags, etc.. Their problem is that they took lots of time to develop in-house, all what later became standard, and almost freely available.

Today, 7 years later, the big talk, new Java products, be it Spring, or JBoss Seam, are still all about the same ideas as in the first J2EE. Spring has the freedom to include whatever technology, but in the end, their scope closely matches J2EE scope. Service Oriented Architecture (SOA), the current buzzword, will come out relatively naturally with J2EE, session EJBs being natural candidate to expose functionalities to the outer world. JBoss (or Geronimo) architecture is fully SOA: a small kernel managing services, that are exposed for other to use.

To learn about EJBs, I still find Enterprise JavaBeans by Richard Monson-Haefel, the best book. The chapter (in the 2nd Edition of the book, not in the 4th unfortunately) where he rolls his own distributed objects system is for me a testimony on how well he wants you to understand what's behind EJBs. And if you want to better understand what's behind RMI, Java Distributed Computing (O'Reilly) has a chapter where the author rolls his own RMI.

To conclude, Sun was really clever in defining J2EE scope and branding several Java technologies as J2EE (despite the awful version naming). While the ideas were very good, the developer's life was forgotten. It was too much a pain in the a** and not flexible enough (especially the ORM side). JBoss gets it right, what people need is a way to use pieces of J2EE, without imposing too much unnecessary hurdle on development.

Job Hunting: Interview Madness

Fri, 04 Nov 2005 10:40:00 +0000

I have been doing interviews those past weeks. Today I had to present my resume for the N time where N is big. It is very boring. In France, it is common to have 5 interviews for a small company (that says a bit about how people trust each other and makes me envy Germany where the practice is to select very strictly on the resume and be more relax about interviews). But all those interviews give a valuable insight on the current market, and on some companies.

Does Eclipse Rot The Brain?

Thu, 27 Oct 2005 17:42:00 +0000

Slashdot presented a really insightful article on Visual Studio by Charles Petzold called Does Visual Studio Rot the Mind. Interestingly it seems that Eclipse does things better than the future Visual Studio.

Autocomplete in Eclipse is really good and does not suffer of the bottom-up problem M. Petzold mentioned, nor from the forced CTRL-Z.
There is no Form designer, and in Java it has been the practice to lay out forms programmatically. The Java Layouts have always been scalable, and Java programs very rarely rely on pixels.
There is no unnecessary import statements.

His positive feedback on XAML is interesting, especially since he thinks this allows for a better automatic UI.

Orkut Statistics and Blog Evolution

Tue, 18 Oct 2005 13:03:00 +0000

I was just having a look at orkut again. As in my last look in the past year, there is not much new or interesting. The technical forums are mostly uninteresting and it almost looks like nothing is going on. Take a look at the orkut java forums, in 1 year, very few messages are good.

Compare that to javablogs.com blogs aggregation, every week there are many interesting posts. The blog model is good because it is targeted at people who want to write. When you blog, you build a history. Forums don’t let you do that. You could have a social model on top of blogs, and I am sure this will be one of their next evolution.

Spring Books Roundup Updated

Mon, 10 Oct 2005 16:27:00 +0000

I am completing my Spring Books Roundup with 2 new books:

Pro Spring, by Apress: a lengthy book, with some unnecessary information, but good overall. I did not like the Part 1, there is not much content in it. Fortunately, Part 2 is much better, dependency injection is very well described, almost on par, with the Fowler article of reference. Other Spring areas are well covered, in a similar manner as in Wrox book, except Java Server Faces, absent from the book, and nothing on Swing as well
Spring Reference Documentation, from the springframework.org website: I should have started with this one, it is of very good quality, and more up-to-date. There is notably a very good chapter on “Source Level Metadata Support” even if I don’t find particularly compelling that they chose yet another abstraction behind Java 5 annotations, partly to support older JDKs. Another plus is a chapter on JMX support, this was lacking in other books.

Interestingly most other books are very similar to the reference documentation, in their presentation and content. The reference documentation is extremely well written. Each time the author carefully explains what are the advantages of the Spring solution, or what does a particular feature solve. Only dependency injection and swing are not treated properly.

Spring Books Roundup

Mon, 10 Oct 2005 11:15:00 +0000

Spring is hip these days, so I decided to learn a bit more about it. I had used Avalon a while ago, I was attracted by its design by component and the way it seemed to lay out a proper infrastructure to build a server application. In the end, I was a bit disappointed, it required a bit too much glue code for my taste and did not provide that much in exchange. I don’t remember if, at that time, it was advertised as IoC (Inversion Of Control) container. This experience is one of the reasons why I did not jump on the Spring ship.

Interesting Plug-In Framework - DPML Transit

Thu, 29 Sep 2005 15:20:00 +0000

Today, I just found out about DPML Transit, it is a small framework that helps you build plug-ins based software. It seems to work a bit with DPML Magic, their build system based upon Ant. Both are quite interesting, since in big projects, you often end up with a packaging per component (which DPML Magic seems to make very simple) and a versioning of those components. DPML Transit allows then for an efficient way to look up a particular version of one component.

Java Puzzlers - Can you figure this out?

Wed, 28 Sep 2005 16:04:00 +0000

The book Java Puzzlers is quite good. I don't think anyone can get every puzzle right. This shows again how you can very easily make someone fail interviews if you ask too silly questions. I suppose that if people were asking those questions they would not expect the right answers, but study the candidate reactions.

Here is a sample:

public class DosEquis { public static void main(String[] args) { char x = 'X'; int i = 0; System.out.print(true ? x : 0); System.out.print(false ? i : x); } }

This will output "X88". Obviously this is not good code, which is precisely one of the book objectives: to show how bad some practices can be. But at the same time you learn a bit more about the Java language and its possibilities. In the latter chapters they have more interesting puzzles.

Is Prolog Better Suited Than SQL?

Mon, 26 Sep 2005 15:36:00 +0000

I am currently reading a Prolog book Artificial Intelligence Through Prolog, I have been doing a bit of Prolog when I was very young and wanted to refresh my memory a bit. It is a very interesting read, especially when I take the viewpoint of our current application where no ACID compliance is required.

It seems to me that all the logic we coded to parametrize SQL queries and construct them dynamically could have been avoided if we had chosen Prolog as Prolog expressions would have been very natural to use in our project. With Prolog, there is no need to think about joins, type of joins, SQL syntax. It is at the level just higher. I wonder very much why Prolog did not become more mainstream as it seems to solve some problems in a much nicer, natural way.

When California Was An Island

Wed, 21 Sep 2005 19:01:00 +0000

This is from an old (1680s) big (2 tons) Coronelli globe, currently displayed in Grand Palais, Paris. Can you spot San Francisco there?

Currying - A Very Interesting Use Of Generics

Fri, 16 Sep 2005 14:41:00 +0000

In bamboo-dht, a distributed java hashtable project, the main developer, Sean C. Rhea, advocates the use of Curries and Thunks (of LISP and ML). He wrote an interesting and valuable document, Async Tutorial, presenting a use of it. I will present the concept here shortly:

public interface Thunk1<T> { void run (T t ) ; } Thunk1<Integer> intThunk = new Thunk1<Integer>() { void run (Integer i ) { System.out.println( i ) ; } } public static <T> Runnable curry(final Thunk1<T> f ,final T t ) { return new Runnable () { public void run ( ) { f.run ( t ) ; } } ; } Runnable print42 = curry( intThunk , new Integer ( 42 ) ) ; print42.run ( ) ; //prints 42 to standard output

So it is a very clever way to have a callback in java. Very clever because you can declare your code to take only a run(), and pass any parameter in it by using a curry. It is very simple to use, even if the small framework around it can scare some people.

However neat the idea is, I am not sure it is practically useful. Sean C. Rhea used that because he did not want to use "unnecessary" instance variables. Ok, but the curry is still creating another instance (actually 2 but one could be static), is the overhead of a class instance without variables that much more than one with variables? And there is another way, without the Curry framework:
Runnable print42 = new Runnable({ public void run() {intThunk.run(new Integer(42));}});
print42.run();

This is actually exactly what the Curry framework would do behind the scenes. The Curry framework makes it a bit more elegant, but I am not sure if that's really more readable for most programmers. I would personally advise the traditional way, use instance variables.

On J2EE Portability Accross Application Servers

Tue, 13 Sep 2005 16:50:00 +0000

Brian Alexander Lee wrote:

“dismal interview where I asked about the candidate’s experience with porting a J2EE application from WebLogic to WebSphere (which was listed on his resume). The candidate said that it was very easy and he just deployed his application with no problems or changes necessary, he had no changes made for the application to run properly. This was one of many bad signs for the candidate.”

I disagree. I would even use his example to show that the portability game is more about configuration and packaging rather than about anything else so that many developers would in-deed not see a big deal into porting an application to a new application server.

JavaBlogs Weekly Top 10 and Java HTML parsing

Mon, 12 Sep 2005 16:00:00 +0000

I took some time to continue my little JavaBlogs analysis, I now have a page summarizing the top 10 most read blog entries in the last week. The page is generated every 24h (this is why there is no ‘best progression’ as of today).

I also fixed some bugs related to HTML in RSS2. I understand a bit better why a RSS 1.0 co-author decided to remove the possibility of HTML descriptions for RSS 3.0. It often does not make sense to keep all that information about styles, fonts, etc. from different sources. What I do is rewrite the HTML, allowing only b,i,a,p,br tags, with the style information stripped. I found the open-source htmlparser java library quite helpful to achieve that.

Spam In Blog Comments

Sat, 10 Sep 2005 01:34:00 +0000

I was a victim like many other of spams in comments. It’s stupid for people to do that on Blogger.com since the links on comments can not be referenced by search engines (they have some special ‘relative’attribute for that) and improve pagerank.

Fortunately Blogger.com provides a word verification step if you want to avoid random spam. However I am a bit disappointed that they force Blogger.com users to do that word verification as well. This time I find it stupid from Blogger.com. They have control on their users, so they could ban spamming users, and for everybody else on Blogger.com, this would be just one less step. I am always a bit annoyed at measures that solve a problem caused by a hand of people by making it more annoying for the majority.

JavaBlogs Daily Analysis

Fri, 09 Sep 2005 17:10:00 +0000

I was wondering what blog entries were the most interesting on Javablogs. I decided to write a small application to do that. It was not much more complex to put it online for others to look at as well. It is currently running on http://gopix.net:8081/javabuzz

It also presents Javablogs a bit differently (I like it better that way).

Please note that it is just the result of a 1 (full) day of work currently. I hopefully will have a bit of time to improve it. For example I’d like to add some graphs about popularity, some weekly stats, and comments in blog entries.

Commons-Beanutils Is Slow

Wed, 07 Sep 2005 17:28:00 +0000

The BeanUtil.popupate(bean,map) can be very handy, but if you care about performance, it is quite slow. I ran a micro benchmark on my machine (centrino 1.8ghz, JDK1.5) and found out that BeanUtils is up to 40x slower than a hand coded solution (where I assign each bean field manually). I was a bit surprised to find such a difference. I suppose there is a big penalty for using reflection and another big one for the BeanUtils abstraction (automatic casting, etc.). I did another test without BeanUtils, using if/else statements vs HashMap.get and found out that the if/else string.equals(…) statements can degrade performance by about 10x. The HashMap appears to be very performant, even with just a few elements in.

Generate your RSS feed in Java

Mon, 05 Sep 2005 14:03:00 +0000

There are some open source projects that can help you in generating or reading RSS feeds in Java. I found only two libraries a bit mature, other code is often embedded in other open source products (jroller for example):

Informa: Does various RSS formats and Atom 0.3. Documentation is better than its alternative, but less focused (has some hibernate helper thingy, some lucene helper, etc.).
Sandler: There is no working homepage while I am writing this. But the code is of decent quality, supports Atom 0.3 and RSS 1.0. It is easy to use it. However in reality it is not much more than a wrapper around some XML parser specialized in generating an RSS structure or an Atom structure.
Ooops, I forgot another important one, Rome. This RSS/Atom framework with a catchy name is very similar to Informa, has good documentation and good looking code. Under the hood it makes use of jdom.

I personally use dom4j since I only need to generate RSS, and RSS, or Atom are just XML. I don't find it particularly verbose to use dom4j for that, and it is very flexible.

If you need to parse feeds, then those libraries might make sense and save you a bit of time. For generating, I think their main interest is to abstract you from the differences in formats. So if you need to handle different formats, a framework will allow you to do it through only one API, which can be a big time-saver.

The Evil Port 80

Thu, 01 Sep 2005 13:46:00 +0000

I was writing an Atom feed generator for my current project. I chosed to support Atom 1.0 since it looks like it has the capabilities to establish as the next standard. Unfortunately I quickly saw that it was quite hard to test it in the real world (out of the good feedvalidator), as almost nobody seems to accept Atom 1.0 feeds yet, even if it is rapidely changing (there is support for it in Firefox CVS version).

So I decided to support RSS as well, the big question was: which RSS version? After grabbing lots of info on the subject, I opted for 1.0 again (more flexible, more different than Atom). It was actually quick to support RSS, but then when in real world, neither Google Desktop nor My Yahoo was willing to accept my feed. I looked at every bit of my xml, fiddled with Tomcat configuration in any possible way when I saw that no request was coming to my server from Yahoo or Google. And finally I thought, hmm maybe it's the port. I restarted my server on port 80, and yup, it worked!

I wonder why Google Desktop and My Yahoo don't support another port than port 80 for RSS feeds.

Google Sidebar Hotkey Activation

Thu, 25 Aug 2005 14:35:00 +0000

I like the new Google Desktop with the sidebar. It shares similarities with Konfabulator, recently bought by yahoo. They both allow easy access to some custom little widgets that I would call "active". They are active because they are refreshed periodically with new information (processor usage, news, scratch pad, emails, etc). But while Konfabulator choose to emphasize on visual effects, Google prefers a more standard information presentation. This shows as well in their choice of technologies:

Google Sidebar plug-ins are just Windows appz that can take advantage of Google Interfaces. That makes them quite powerful in theory, but programming them is less accessible.
Konfabulator plug-ins are Javascript+XML, the Javascript is not just regular client-side javascript, it can use Konfabulator API (containing many effect and rudimentary network access), and COM objects . That makes them very focused on presentation, and the Internet.

In the long run, Google choice makes sense, the forthcoming Avalon will make visual effects very accessible to windows developers.

Now back to the subject, I missed the 'activation on hotkey' feature from Konfabulator for the Google Sidebar. Fortunately, I a have found a powerful little open-source program, Autohotkey, that allowed me to do that very quickly. Here is the script I use (it's a hack since it relies on toolbar size (but not position), but I like the default position and it works (only using floating deskbar, i let you figure out for the non floating version)):
F12::
MouseGetPos, X, Y
if WinExist("ahk_class _GD_Sidebar")
{
WinActivate
BlockInput,On
MouseClick, left, 160, 16
MouseMove, X,Y
BlockInput,Off
return
}
else if WinExist("ahk_class ATL:0044A4C8")
{
WinActivate
BlockInput,On
MouseClick, left, 158, 16
MouseMove, X,Y
BlockInput,Off
return
}

I Need Another DB Framework!

Tue, 23 Aug 2005 19:37:00 +0000

I am currently facing a problem that neither Hibernate nor iBatis solves nicely. I also looked at other ORM or just DB framework, without success.

What I would need is a framework that generates PreparedStatements with a query by Criteria like API. I have many queries that are similar but varying according to different input parameters. iBatis can handle this, but for complex queries and scenarios, the XML becomes completely unreadable, and you therefore loose any advantage that iBatis was bringing with the externalization of SQL statements in XML. The other issue I have with using iBatis is that for another part of my project, the automatic generation of SQL statements a-la Hibernate is useful. Hibernate has a very nice Query by Criteria API, but it lacks just a tiny bit of flexibility in customizing queries. For example, I could not find a way to specify a “USE INDEX(index_name)” in the generated SQL, after the SELECT FROM xxx and before the rest of the query. I did not find either a way to specify the use of a “STRAIGHT_JOIN” instead of an INNER JOIN. These are all MySQL specific issues, but those little things are extremely useful at improving some of my queries performances. Writing N sql queries hard coded is not a good option, since this N can be quite big, which is why I am using Query by Criteria in the first place.

del.icio.us toolbar customized

Sun, 21 Aug 2005 05:20:00 +0000

When I bookmark articles with delicious, I like to keep the content on my hard drive, because pages sometimes change, or are removed, or I want to do local searches. I believe this is one reason some people like furl (furl keeps a copy on their server that only yourself can read, but does not allow search).

A combination of slogger and delicious could solve partially the problem. But it is not integrated, I can't get my local version from delicious, so I loose the tagging, listing and all other plus from delicious.

I added my own feature to the delicious toolbar, which I like very much. This new toolbar saves automatically the file you bookmark (on the + button), and will add a link in your delicious home to the local version (if it exists).

I have it publicly accessible at http://perso.wanadoo.fr/logos0 1/deltoolbar.html

It is not meant to be used by everybody as it is not official. But the page will give you an idea of what it does. If you think it is useful, I will improve it, otherwise it will stay the way it is because it fits my use.

Categories: delicious, firefox, toolbar

Inside the Java Virtual Machine

Thu, 18 Aug 2005 11:03:00 +0000

I am reading an old book, Inside the Java Virtual Machine. Some old books don't age, and this is one of them. The chapter on the Java Virtual Machine is just excellent and should be read by every Java developer. It explains each step a JVM does when you run a Java program, very clearly.

You could get plenty of stupid interview questions from it like: How is the Java stack used? Between method area, heap, pc register, stack which one are shared among threads?

Also they saw the full potential of Java quite early on (1997). They explain how the JVM specs allow for very different implementations, ones that can run in different environments, for example, simplifying a bit: low memory, embedded world, or lots of memory, mainframe world. It is not an accident if Microsoft chosed a very similar design for the CLI of .NET, they have been looking for getting into the embedded area for quite some time, and apparently, they are making good progress.

Firefox Shortcut URL as Command Line for the Web

Tue, 16 Aug 2005 13:42:00 +0000

I have been doing some work on Firefox extensions. Mozilla/Firefox is a really interesting world, it is extremely flexible, you can make it transform the web the way you want.

Recently, someone started YubNub, the "command line for the web", a very simple idea, but useful. This is a bit similar to the Firefox Search Bar, except more powerful, and quicker to use. Mozilla developers actually had that idea a long time ago (around 2000), it is called shortcut url. Basically to any bookmark, you can add a shortcut, you then just have to type that shortcut (let say g for google.com) in the address bar to go to the address pointed by the shortcut. This shortcut accepts 1 parameter (represented by %s in the URL), so you can do "g chasethedevil" to search google for "chasethedevil" if you set up your shortcut properly. Firefox can build this dynamic URL automatically for you if in an input box, you right click and choose "Add a keyword for this search".

Now we have seen the Firefox address bar can be used to quickly perform a search on a selected website. But this can be much more powerful. As this page suggests, you can associate a bookmarklet with a shortcut, you can execute whatever javascript you wish. One very simple example is:
javascript:void(location.href= 'http://web.archive.org/web/*dc_/'+location.href). This shows how you can use the address bar in a very flexible manner. It would be easy to support a multiple parameters search using a bookmarklet. There is a whole list of commands very similar to yubnub ones in addition to those on the previously mentioned link.

I first thought about writing a simple extension to have my command line in Firefox (for example by extending the getShortcutOnURI function), but after some information gathering, I discovered it was already possible. Firefox is truly amazing. It is a bit of a shame that this functionality is not a bit better exposed to the user.

tags: Firefox shortcut extension bookmarklet

Hibernate Criteria API vs HQL Overhead

Wed, 10 Aug 2005 14:16:00 +0000

I was wondering what was the framework weight in the performance of my application, and I wanted to check at several frameworks performance. I used JMeter to benchmark a stripped application (database access through tomcat) under various loads. My very modest test shows no significant difference between Hibernate Criteria API and HQL access for queries. Criteria might be a few milliseconds slower, but my query time will take 10x more time, even for a relatively simple query (my query is has 3 inner joins, is grouped, with a count and takes only 4ms when performed once in mysql under no load). It seems much more important for the Framework of choice to provide good scalability, easy development and maintenance, rather than saving a few cycles, unless the Framework becomes a bottleneck.

How to become a JavaRSS member on blogger.com

Fri, 05 Aug 2005 13:32:00 +0000

The trick is to create a post with the javarss image and to use the link generated by blogger.com to display the image. Then you can follow the rules from http://www.javarss.com/pages/?page=suggest

Java is more productive than Ruby/Rails

Tue, 26 Jul 2005 13:26:00 +0000

I have been doing some Ruby On Rails, for 2 small projects. While I think it is good, I think it is overhyped as well. It is well designed, has good ideas (easy configuration), and focus on the right problem, architecture. But my conclusion is that I am not more productive with it than with Java.

I think most of the development time is not spent coding, but thinking. It is a very obvious statement, and yet too often ignored.

féminisme et statistiques

Thu, 21 Jul 2005 12:16:00 +0000

Sur ce lien, un article un peu trop long, mais avec pas mal de points intéressants:

6 femmes par mois décèdent, en France, par violence conjugale, soit plus que le nombre de victimes du metro anglais sur un an. Pour toutes les statistiques sur les chances de mourir suivant les causes, aux Etats-Unis, cliquer ici (on voit notamment que le plus grand risque est l'accident de voiture, ou la chute, ou le suicide par arme à feu, devant les attaques par arme à feu).
Un petit rappel sur l'antiféminisme chrétien, les femmes y étant qualifiées: de «défectuosité naturelle», de «raté de la création», d’«animal inquiet», de «mouche éphémère», de «méchante bête», de «porte du Diable», de «sac de fiente», de «diable domestique», de «racine du Mal», de «rejeton de tous les vices», de «piège tendu par l’ennemi», de «terre puante», d’«appendice de la race humaine». Ce qui lui permet d'ajouter: «Au lieu d’expliquer la misogynie des hommes par celle des religions, ne faudrait-il pas plutôt expliquer la misogynie des religions par celle des hommes?».
Une citation intéressante sur le voile en France: «ces filles qui affichent leur voile dans une société laïque sont le pendant exact de celles qui le refusent dans une société musulmane» laisse à réfléchir; et aussi un développement sur Theo Van Gogh dépeint ici en tant que raciste, misogyne et opportuniste, un avis que je ne partage pas.

C'est un peu extrême par moments, mais souvent juste. Sur le même sujet, Le Monde avait un petit article sur les wagons réservés aux femmes à Tokyo. Bon, j'ai peut-être un peu pété les plombs avec toute cette lecture.

Tags: feminism francais actualités.

Ruby, Python, JavaScript, Lua, Java, C++ benchmark

Wed, 20 Jul 2005 11:49:00 +0000

Simple benchmark but with interesting comments here:

Some of the conclusions are:

Java interpreter is very good
Java JIT is as fast as C++
Ruby, JavaScript, Python are of similar order of magnitude speed, Ruby the slowest.

It would have been interesting to benchmark interpreted languages for the JVM.

Now for most projects, architecture, not language is key in achieving good performance.

Fowler On Your Future: editing the abstract representation of a program

Tue, 19 Jul 2005 12:40:00 +0000

This Fowler article on "language workbench" reminds me of Gosling Jackpot ideas: it is about a different interaction with your program where the source code does not matter so much, but its structure. It must be fascinating to work on this kind of project.

Maybe MDA sounds similar to you, after all, actual language workbench systems are generating source code. But the fundamental difference is that a workbench generated program is much richer and can capture everything you want to do (MDA is too based on UML to be that powerful), also, the source could disappear and be replaced by byte codes only.

Now this all seems very nice, but unfortunately, there are always problems, and Fowler does not miss them: vendor lock-in (not that big of a deal if source is generated), integration with dev tools (especially version control), maturity. It looks like an open source language workbench would help here.

It is interesting to see that Intellij (yes, the company behind IDEA) is already well on its way. You can check it out in Fowler examples. His examples are a bit too simple and not very far off rules in rules engines, but the language definition tool does look very neat.

As Rainer Joswig shows in his video, LISP seems like a particularly good language to do this. This is hardly surprising since LISP is very near the abstract tree structure and therefore already very used in the AI circles.

Tags: java programming

Jython instead of Ant: sample code

Sat, 16 Jul 2005 01:50:00 +0000

Here is an example of how to use jython to deploy a web application under tomcat. The application is supposed here to be WAR packaged in a 'web' directory. Libraries used for the web-app are already in web/WEB-INF/lib, classes are compiled automatically by Eclipse under web/WEB-INF/classes, and resources are copied automatically by Eclipse. If necessary, it is very easy to script those actions in the python script, with very little code.

The longest code is the TreeCopier class, that should probably be part of any python make script. The reload method could be reworked to provide a generic method for reloading a tomcat web application. The following is just a working sample code as proof of concept.


# call jython make.py method1 method2, ... 
# to invoke method1 and then method2, ...

import sys, os, os.path, shutil, urllib, base64, re
web_dir = 'web'
webapp_name = 'myrestaurant'
tomcat_dir = 'c:/java/Tomcat 5.5'

#copy web directory to tomcat app dir
def deploy():
   target_dir = tomcat_dir+'/webapps'+'/'+webapp_name
   copier=LatestTreeCopier()
   copier.copytree(web_dir,target_dir)
   print "copied %s file(s) to %s" % (copier.count,target_dir)

#reload tomcat web application
def reload():
   url = "http://localhost:8080/manager/reload?path=/"+webapp_name
   base64string = base64.encodestring("%s:%s"%('michelin','michelin'))[:-1]
   opener = urllib.URLopener()
   opener.addheader('Authorization',"Basic %s"%base64string)
   h = opener.open(url)
   print h.read()
   h.close()

class TreeCopier:
   def __init__(self):
       self.count = 0

   def filter(self, srcname, dstname):
       return true

   def filterDir(self, srcname):
       return true

   def copytree(self, src, dst, symlinks=0):
       """Recursively copy a directory tree using copy2().
        """
       names = os.listdir(src)
       if not os.path.exists(dst):
           os.mkdir(dst)
       count = 0
       for name in names:
           srcname = os.path.join(src, name)
           dstname = os.path.join(dst, name)
           try:
               if symlinks and os.path.islink(srcname):
                   linkto = os.readlink(srcname)
                   os.symlink(linkto, dstname)
               elif os.path.isdir(srcname):
                   if self.filterDir(srcname):
                       self.copytree(srcname, dstname, symlinks)
               else:
                   if self.filter(srcname, dstname):
                       shutil.copy2(srcname, dstname)
                       print srcname
                       count += 1
           except (IOError, os.error), why:
               print "Can't copy %s to %s: %s"%('srcname', 'dstname', str(why))
       self.count += count

class LatestTreeCopier(TreeCopier):
   def __init__(self):
       self.count = 0
       self.excludePattern = re.compile(
               '(^\.svn.*|.*\.swp$|.*\.bak$|.*~$|.*\.swo$)',re.I)
       self.excludeDirPattern = re.compile('^\.svn.*',re.I)

   def filter(self, srcname, dstname):
       return (not self.excludePattern.match(os.path.basename(srcname)))
               and (os.path.getmtime(srcname) > os.path.getmtime(dstname))

   def filterDir(self, srcname):
       return (not self.excludeDirPattern.match(os.path.basename(srcname)))


if __name__ == "__main__":
   if (len(sys.argv)) >= 2:
       methods = sys.argv[1:]
       for m in methods:
           method = m+'()'
           print "launching %s" % method
           eval(method)
   else:
       deploy()
       reload()

Java development without Ant

Tue, 12 Jul 2005 16:16:00 +0000

When Ant came out, I welcomed it. It was better than Makefiles for Java, it solved the path and classpath format difference between Windows and Linux, did not rely on shell programs. It did provide a platform independent way for building Java software. At that time, there was another alternative, JMK which had a lisp like syntax, and solved the same problems as Ant. But Ant was backed by apache, and this were the times of XML everywhere. I got more and more fed up with ant scripts over the years. During a long time you had to resort to an XML trick just to include other ant file, you did not have conditionals, but more importantly, it is quite a pain to write long ant scripts in XML. XML is not that good to write logic with, XSLT did not become as big as it could have been partly for this reason.

lion

Tue, 28 Jun 2005 17:00:00 +0000

test

This is who I am

Sun, 19 Jun 2005 18:20:00 +0000