Discussion:
[TIP,BISECTED] Negative nice values have become big positive numbers
Sitsofe Wheeler
2009-03-08 23:18:50 UTC
Permalink
(Forgot to cc lkml. Resending...)

Formally negative nice values have started become very big in positive
integers in -tip kernels:

2 root 15 2147483647 0 0 0 S 0.0 0.0 0:00.00 kthreadd

I've just finished bisecting down to this commit:

commit 1427cdf0592368bdec57276edaf714040ee8744f
Author: Lai Jiangshan <***@cn.fujitsu.com>
Date: Fri Mar 6 17:21:47 2009 +0100

tracing: infrastructure for supporting binary record

Impact: save on memory for tracing

Current tracers are typically using a struct(like struct ftrace_entry,
struct ctx_switch_entry, struct special_entr etc...)to record a binary
event. These structs can only record a their own kind of events.
A new kind of tracer need a new struct and a lot of code too handle it.

So we need a generic binary record for events. This infrastructure
is for this purpose.

[***@gmail.com: rebase against latest -tip, make it safe while sched
tracing as reported by Steven Rostedt]

Signed-off-by: Lai Jiangshan <***@cn.fujitsu.com>
Signed-off-by: Frederic Weisbecker <***@gmail.com>
Acked-by: Steven Rostedt <***@goodmis.org>
LKML-Reference: <1236356510-8381-3-git-send-email-***@gmail.com>
Signed-off-by: Ingo Molnar <***@elte.hu>

I haven't had a chance to try reverting it but here's the bisect log:

git bisect start
# bad: [8a2b9322fa3f95ab51a863d5095543a32dd9ea8f] Merge branch 'x86/mm-v2'
git bisect bad 8a2b9322fa3f95ab51a863d5095543a32dd9ea8f
# good: [3a450de1365d20afde406f0d9b2931a5e4a4fd6a] x86: UV: remove uv_flush_tlb_others() WARN_ON
git bisect good 3a450de1365d20afde406f0d9b2931a5e4a4fd6a
# good: [e13d8d0e99b8242db9d415f6548378716773fc79] Merge branch 'auto-kmemcheck-next' into auto-latest
git bisect good e13d8d0e99b8242db9d415f6548378716773fc79
# good: [bbfc9e1a1b3d32170cefc3cff3adf0cffe00e48b] manual merge of auto-perfcounters-next
git bisect good bbfc9e1a1b3d32170cefc3cff3adf0cffe00e48b
# good: [6fb755b0f709921e98e50ddf3c7537129e0b11f7] Merge branch 'tracing/core'
git bisect good 6fb755b0f709921e98e50ddf3c7537129e0b11f7
# good: [cd3c1cde262b0f068d27d980648892b2843966d4] Merge git://git.kernel.org/pub/scm/linux/kernel/git/wim/linux-2.6-watchdog
git bisect good cd3c1cde262b0f068d27d980648892b2843966d4
# bad: [902b3c2ce7bb48418da33880df4596cddb07e4e5] Merge branch 'x86/apic'
git bisect bad 902b3c2ce7bb48418da33880df4596cddb07e4e5
# good: [12c0faeaad9f764d4532500c84074b83540f4012] Merge branch 'sched/cleanups'
git bisect good 12c0faeaad9f764d4532500c84074b83540f4012
# good: [0538dc9a21063dae6e1f1e945dec10f781b3c12e] Merge branch 'tracing/ftrace'
git bisect good 0538dc9a21063dae6e1f1e945dec10f781b3c12e
# good: [78ff7fae04554b49d29226ed12536268c2500d1f] x86: implement atomic text_poke() via fixmap
git bisect good 78ff7fae04554b49d29226ed12536268c2500d1f
# bad: [9de36825b321fe9fe9cf73260554251af579f4ca] tracing: trace_bprintk() cleanups
git bisect bad 9de36825b321fe9fe9cf73260554251af579f4ca
# bad: [1ba28e02a18cbdbea123836f6c98efb09cbf59ec] tracing: add trace_bprintk()
git bisect bad 1ba28e02a18cbdbea123836f6c98efb09cbf59ec
# bad: [1427cdf0592368bdec57276edaf714040ee8744f] tracing: infrastructure for supporting binary record
git bisect bad 1427cdf0592368bdec57276edaf714040ee8744f
--
Sitsofe | http://sucs.org/~sits/
Steven Rostedt
2009-03-08 23:49:08 UTC
Permalink
Post by Sitsofe Wheeler
(Forgot to cc lkml. Resending...)
Formally negative nice values have started become very big in positive
2 root 15 2147483647 0 0 0 S 0.0 0.0 0:00.00 kthreadd
Is this the output of top?
Post by Sitsofe Wheeler
commit 1427cdf0592368bdec57276edaf714040ee8744f
Date: Fri Mar 6 17:21:47 2009 +0100
$ git show 1427cdf0592368bdec57276edaf714040ee8744f | diffstat
include/linux/ftrace.h | 3 +
kernel/trace/Kconfig | 6 ++
kernel/trace/Makefile | 1
kernel/trace/trace.c | 56 +++++++++++++++++++++++++++
kernel/trace/trace.h | 12 +++++
kernel/trace/trace_bprintk.c | 87
+++++++++++++++++++++++++++++++++++++++++++
kernel/trace/trace_output.c | 75 +++++++++++++++++++++++++++++++++++++
7 files changed, 240 insertions(+)

I find it hard to believe that this would cause normal nice values to be
messed up. The only file that could could come close to messing with nice
values in top is ftrace.h:


diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h
index 4987694..1c9cdca 100644
--- a/include/linux/ftrace.h
+++ b/include/linux/ftrace.h
@@ -223,6 +223,9 @@ extern int ftrace_make_nop(struct module *mod,
*/
extern int ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr);

+#ifdef CONFIG_TRACE_BPRINTK
+extern int trace_vbprintk(unsigned long ip, const char *fmt, va_list args);
+#endif

/* May be defined in arch */
extern int ftrace_arch_read_dyn_info(char *buf, int size);



And that too would be hard to affect nice values. Do you have a config
that I could test with?

Thanks,

-- Steve
Ingo Molnar
2009-03-09 07:08:24 UTC
Permalink
Post by Steven Rostedt
Post by Sitsofe Wheeler
(Forgot to cc lkml. Resending...)
Formally negative nice values have started become very big in positive
2 root 15 2147483647 0 0 0 S 0.0 0.0 0:00.00 kthreadd
Is this the output of top?
seems so.
Post by Steven Rostedt
Post by Sitsofe Wheeler
commit 1427cdf0592368bdec57276edaf714040ee8744f
Date: Fri Mar 6 17:21:47 2009 +0100
$ git show 1427cdf0592368bdec57276edaf714040ee8744f | diffstat
include/linux/ftrace.h | 3 +
kernel/trace/Kconfig | 6 ++
kernel/trace/Makefile | 1
kernel/trace/trace.c | 56 +++++++++++++++++++++++++++
kernel/trace/trace.h | 12 +++++
kernel/trace/trace_bprintk.c | 87
+++++++++++++++++++++++++++++++++++++++++++
kernel/trace/trace_output.c | 75 +++++++++++++++++++++++++++++++++++++
7 files changed, 240 insertions(+)
I find it hard to believe that this would cause normal nice
values to be messed up. The only file that could could come
Correct - maybe it's these two nearby commits that cause the
problems:

fef20d9: vsprintf: unify the format decoding layer for its 3 users
4370aa4: vsprintf: add binary printf

they do affect generic code. If we broke vsnprintf (which the
nice value output code uses) then that might be a plausible
explanation.

Ingo
Frederic Weisbecker
2009-03-09 08:41:34 UTC
Permalink
Post by Ingo Molnar
Post by Steven Rostedt
Post by Sitsofe Wheeler
(Forgot to cc lkml. Resending...)
Formally negative nice values have started become very big in positive
2 root 15 2147483647 0 0 0 S 0.0 0.0 0:00.00 kthreadd
Is this the output of top?
seems so.
Post by Steven Rostedt
Post by Sitsofe Wheeler
commit 1427cdf0592368bdec57276edaf714040ee8744f
Date: Fri Mar 6 17:21:47 2009 +0100
$ git show 1427cdf0592368bdec57276edaf714040ee8744f | diffstat
include/linux/ftrace.h | 3 +
kernel/trace/Kconfig | 6 ++
kernel/trace/Makefile | 1
kernel/trace/trace.c | 56 +++++++++++++++++++++++++++
kernel/trace/trace.h | 12 +++++
kernel/trace/trace_bprintk.c | 87
+++++++++++++++++++++++++++++++++++++++++++
kernel/trace/trace_output.c | 75 +++++++++++++++++++++++++++++++++++++
7 files changed, 240 insertions(+)
I find it hard to believe that this would cause normal nice
values to be messed up. The only file that could could come
Correct - maybe it's these two nearby commits that cause the
fef20d9: vsprintf: unify the format decoding layer for its 3 users
4370aa4: vsprintf: add binary printf
they do affect generic code. If we broke vsnprintf (which the
nice value output code uses) then that might be a plausible
explanation.
Ingo
Thanks for this report Sitsofe. I have some ideas about the origin of this bug.
Anyway I'm in the middle of moving to another town. There are very good chances
that I send a fix tomorrow but not today.

Thanks.
Sitsofe Wheeler
2009-03-09 12:56:57 UTC
Permalink
Post by Ingo Molnar
Post by Steven Rostedt
Post by Sitsofe Wheeler
Formally negative nice values have started become very big in positive
2 root 15 2147483647 0 0 0 S 0.0 0.0 0:00.00 kthreadd
Is this the output of top?
seems so.
Post by Steven Rostedt
Post by Sitsofe Wheeler
commit 1427cdf0592368bdec57276edaf714040ee8744f
I find it hard to believe that this would cause normal nice
values to be messed up. The only file that could could come
Correct - maybe it's these two nearby commits that cause the
fef20d9: vsprintf: unify the format decoding layer for its 3 users
4370aa4: vsprintf: add binary printf
they do affect generic code. If we broke vsnprintf (which the
nice value output code uses) then that might be a plausible
explanation.
OK I've just rebisected (and reverted) this down to the following:

commit fef20d9c1380f04ba9492d6463148db07b413708
Author: Frederic Weisbecker <***@gmail.com>
Date: Fri Mar 6 17:21:50 2009 +0100

vsprintf: unify the format decoding layer for its 3 users

An new optimization is making its way to ftrace. Its purpose is to
make trace_printk() consuming less memory and be faster.

Written by Lai Jiangshan, the approach is to delay the formatting
job from tracing time to output time.

Currently, a call to trace_printk() will format the whole string and
insert it into the ring buffer. Then you can read it on /debug/tracing/trace
file.

The new implementation stores the address of the format string and
the binary parameters into the ring buffer, making the packet more compact
and faster to insert.
Later, when the user exports the traces, the format string is retrieved
with the binary parameters and the formatting job is eventually done.

The new implementation rewrites a lot of format decoding bits from
vsnprintf() function, making now 3 differents functions to maintain
in their duplicated parts of printf format decoding bits.

Suggested by Ingo Molnar, this patch tries to factorize the most
possible common bits from these functions.
The real common part between them is the format decoding. Although
they do somewhat similar jobs, their way to export or import the parameters
is very different. Thus, only the decoding layer is extracted, unless you see
other parts that could be worth factorized.

Changes in V2:

- Address a suggestion from Linus to group the format_decode() parameters inside
a structure.

Changes in v3:

- Address other cleanups suggested by Ingo and Linus such as passing the
printf_spec struct to the format helpers: pointer()/number()/string()
Note that this struct is passed by copy and not by address. This is to
avoid side effects because these functions often change these values and the
changes shoudn't be persistant when a callee helper returns.
It would be too risky.

- Various cleanups (code alignement, switch/case instead of if/else fountains).

- Fix a bug that printed the first format specifier following a %p

Changes in v4:

- drop unapropriate const qualifier loss while casting fmt to a char *
(thanks to Vegard Nossum for having pointed this out).

Signed-off-by: Frederic Weisbecker <***@gmail.com>
Cc: Linus Torvalds <***@linux-foundation.org>
Acked-by: Steven Rostedt <***@goodmis.org>
LKML-Reference: <1236356510-8381-6-git-send-email-***@gmail.com>
Signed-off-by: Ingo Molnar <***@elte.hu>

Here's the new bisect log:
# bad: [546e5354a6e4ec760ac03ef1148e9a4762abb5f5] Merge branch 'core/printk' into tracing/ftrace
# good: [78ff7fae04554b49d29226ed12536268c2500d1f] x86: implement atomic text_poke() via fixmap
git bisect start '546e5354a6e4ec760ac03ef1148e9a4762abb5f5' '78ff7fae04554b49d29226ed12536268c2500d1f'
# good: [16097439703bcd38e9fe5608c12add6dacb825ea] Merge branches 'tracing/ftrace' and 'tracing/function-graph-tracer' into tracing/core
git bisect good 16097439703bcd38e9fe5608c12add6dacb825ea
# good: [4370aa4aa75391a5e2e06bccb0919109f725ed8e] vsprintf: add binary printf
git bisect good 4370aa4aa75391a5e2e06bccb0919109f725ed8e
# good: [bc722f508a5bcbb65a7bb0c7ce8e3934f5763a1a] Merge branch 'tip/tracing/ftrace' of git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-2.6-trace into tracing/ftrace
git bisect good bc722f508a5bcbb65a7bb0c7ce8e3934f5763a1a
# bad: [fef20d9c1380f04ba9492d6463148db07b413708] vsprintf: unify the format decoding layer for its 3 users
git bisect bad fef20d9c1380f04ba9492d6463148db07b413708

Apologies for getting it wrong the first time round (I did say I hadn't
actually reverted the commit though ;)... The problem is that doing
blind bisection actually takes a large amount of time along with plenty
of reboots (additionally the makefile changed enough that oldconfig need
to have prompts answered in many cases) and I was trying to go as fast
as possible (I found the problem late in the evening and wanted to send
something before I fell asleep). Is there an IRC channel people testing
-tip hang out in? I tried #fedora-devel and #fedora-kernel but most of
the folks there were asleep or not testing -tip kernels.
--
Sitsofe | http://sucs.org/~sits/
Sitsofe Wheeler
2009-03-09 08:39:27 UTC
Permalink
Post by Steven Rostedt
Post by Sitsofe Wheeler
Formally negative nice values have started become very big in positive
2 root 15 2147483647 0 0 0 S 0.0 0.0 0:00.00 kthreadd
Is this the output of top?
Yes.
Post by Steven Rostedt
Post by Sitsofe Wheeler
commit 1427cdf0592368bdec57276edaf714040ee8744f
Date: Fri Mar 6 17:21:47 2009 +0100
I find it hard to believe that this would cause normal nice values to be
messed up. The only file that could could come close to messing with nice
And that too would be hard to affect nice values. Do you have a config
that I could test with?
See below:

# Automatically generated make config: don't edit
# Linux kernel version: 2.6.29-rc6
# Mon Feb 23 18:01:55 2009
#
# CONFIG_64BIT is not set
CONFIG_X86_32=y
# CONFIG_X86_64 is not set
CONFIG_X86=y
CONFIG_ARCH_DEFCONFIG="arch/x86/configs/i386_defconfig"
CONFIG_GENERIC_TIME=y
CONFIG_GENERIC_CMOS_UPDATE=y
CONFIG_CLOCKSOURCE_WATCHDOG=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_HAVE_LATENCYTOP_SUPPORT=y
CONFIG_FAST_CMPXCHG_LOCAL=y
CONFIG_MMU=y
CONFIG_ZONE_DMA=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y
CONFIG_GENERIC_BUG=y
CONFIG_GENERIC_HWEIGHT=y
CONFIG_ARCH_MAY_HAVE_PC_FDC=y
# CONFIG_RWSEM_GENERIC_SPINLOCK is not set
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_ARCH_HAS_CPU_IDLE_WAIT=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
# CONFIG_GENERIC_TIME_VSYSCALL is not set
CONFIG_ARCH_HAS_CPU_RELAX=y
CONFIG_ARCH_HAS_DEFAULT_IDLE=y
CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
# CONFIG_HAVE_SETUP_PER_CPU_AREA is not set
# CONFIG_HAVE_CPUMASK_OF_CPU_MAP is not set
CONFIG_ARCH_HIBERNATION_POSSIBLE=y
CONFIG_ARCH_SUSPEND_POSSIBLE=y
# CONFIG_ZONE_DMA32 is not set
CONFIG_ARCH_POPULATES_NODE_MAP=y
# CONFIG_AUDIT_ARCH is not set
CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_X86_BIOS_REBOOT=y
CONFIG_KTIME_SCALAR=y
CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"

#
# General setup
#
CONFIG_EXPERIMENTAL=y
CONFIG_BROKEN_ON_SMP=y
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_LOCALVERSION=""
CONFIG_LOCALVERSION_AUTO=y
# CONFIG_SWAP is not set
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
CONFIG_POSIX_MQUEUE=y
# CONFIG_BSD_PROCESS_ACCT is not set
# CONFIG_TASKSTATS is not set
# CONFIG_AUDIT is not set

#
# RCU Subsystem
#
CONFIG_CLASSIC_RCU=y
# CONFIG_TREE_RCU is not set
# CONFIG_PREEMPT_RCU is not set
# CONFIG_TREE_RCU_TRACE is not set
# CONFIG_PREEMPT_RCU_TRACE is not set
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
CONFIG_LOG_BUF_SHIFT=17
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
# CONFIG_GROUP_SCHED is not set
# CONFIG_CGROUPS is not set
# CONFIG_SYSFS_DEPRECATED_V2 is not set
CONFIG_RELAY=y
# CONFIG_NAMESPACES is not set
CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE=""
# CONFIG_CC_OPTIMIZE_FOR_SIZE is not set
CONFIG_SYSCTL=y
CONFIG_EMBEDDED=y
# CONFIG_UID16 is not set
# CONFIG_SYSCTL_SYSCALL is not set
CONFIG_KALLSYMS=y
CONFIG_KALLSYMS_ALL=y
# CONFIG_KALLSYMS_EXTRA_PASS is not set
CONFIG_HOTPLUG=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
# CONFIG_PCSPKR_PLATFORM is not set
# CONFIG_COMPAT_BRK is not set
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_ANON_INODES=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
CONFIG_SHMEM=y
CONFIG_AIO=y
CONFIG_VM_EVENT_COUNTERS=y
CONFIG_PCI_QUIRKS=y
CONFIG_SLUB_DEBUG=y
# CONFIG_SLAB is not set
CONFIG_SLUB=y
# CONFIG_SLOB is not set
# CONFIG_PROFILING is not set
CONFIG_TRACEPOINTS=y
CONFIG_MARKERS=y
CONFIG_HAVE_OPROFILE=y
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
CONFIG_HAVE_IOREMAP_PROT=y
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
CONFIG_HAVE_ARCH_TRACEHOOK=y
CONFIG_HAVE_GENERIC_DMA_COHERENT=y
CONFIG_SLABINFO=y
CONFIG_RT_MUTEXES=y
CONFIG_BASE_SMALL=0
# CONFIG_MODULES is not set
CONFIG_BLOCK=y
# CONFIG_LBD is not set
CONFIG_BLK_DEV_IO_TRACE=y
# CONFIG_BLK_DEV_BSG is not set
# CONFIG_BLK_DEV_INTEGRITY is not set

#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
# CONFIG_IOSCHED_AS is not set
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
# CONFIG_DEFAULT_AS is not set
# CONFIG_DEFAULT_DEADLINE is not set
CONFIG_DEFAULT_CFQ=y
# CONFIG_DEFAULT_NOOP is not set
CONFIG_DEFAULT_IOSCHED="cfq"
CONFIG_FREEZER=y

#
# Processor type and features
#
CONFIG_TICK_ONESHOT=y
CONFIG_NO_HZ=y
CONFIG_HIGH_RES_TIMERS=y
CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
# CONFIG_SMP is not set
# CONFIG_SPARSE_IRQ is not set
# CONFIG_X86_MPPARSE is not set
CONFIG_X86_PC=y
# CONFIG_X86_ELAN is not set
# CONFIG_X86_VOYAGER is not set
# CONFIG_X86_GENERICARCH is not set
# CONFIG_X86_VSMP is not set
# CONFIG_X86_RDC321X is not set
# CONFIG_SCHED_OMIT_FRAME_POINTER is not set
# CONFIG_PARAVIRT_GUEST is not set
# CONFIG_MEMTEST is not set
# CONFIG_M386 is not set
# CONFIG_M486 is not set
# CONFIG_M586 is not set
# CONFIG_M586TSC is not set
# CONFIG_M586MMX is not set
# CONFIG_M686 is not set
# CONFIG_MPENTIUMII is not set
# CONFIG_MPENTIUMIII is not set
CONFIG_MPENTIUMM=y
# CONFIG_MPENTIUM4 is not set
# CONFIG_MK6 is not set
# CONFIG_MK7 is not set
# CONFIG_MK8 is not set
# CONFIG_MCRUSOE is not set
# CONFIG_MEFFICEON is not set
# CONFIG_MWINCHIPC6 is not set
# CONFIG_MWINCHIP3D is not set
# CONFIG_MGEODEGX1 is not set
# CONFIG_MGEODE_LX is not set
# CONFIG_MCYRIXIII is not set
# CONFIG_MVIAC3_2 is not set
# CONFIG_MVIAC7 is not set
# CONFIG_MPSC is not set
# CONFIG_MCORE2 is not set
# CONFIG_GENERIC_CPU is not set
# CONFIG_X86_GENERIC is not set
CONFIG_X86_CPU=y
CONFIG_X86_CMPXCHG=y
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_X86_XADD=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INVLPG=y
CONFIG_X86_BSWAP=y
CONFIG_X86_POPAD_OK=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_X86_TSC=y
CONFIG_X86_CMOV=y
CONFIG_X86_MINIMUM_CPU_FAMILY=4
CONFIG_X86_DEBUGCTLMSR=y
CONFIG_PROCESSOR_SELECT=y
CONFIG_CPU_SUP_INTEL=y
# CONFIG_CPU_SUP_CYRIX_32 is not set
# CONFIG_CPU_SUP_AMD is not set
# CONFIG_CPU_SUP_CENTAUR_32 is not set
# CONFIG_CPU_SUP_TRANSMETA_32 is not set
# CONFIG_CPU_SUP_UMC_32 is not set
CONFIG_X86_DS=y
CONFIG_X86_PTRACE_BTS=y
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
CONFIG_DMI=y
# CONFIG_IOMMU_HELPER is not set
# CONFIG_IOMMU_API is not set
CONFIG_NR_CPUS=1
# CONFIG_PREEMPT_NONE is not set
CONFIG_PREEMPT_VOLUNTARY=y
# CONFIG_PREEMPT is not set
CONFIG_X86_UP_APIC=y
CONFIG_X86_UP_IOAPIC=y
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_REROUTE_FOR_BROKEN_BOOT_IRQS=y
CONFIG_X86_MCE=y
CONFIG_X86_MCE_NONFATAL=y
CONFIG_X86_MCE_P4THERMAL=y
CONFIG_VM86=y
# CONFIG_TOSHIBA is not set
# CONFIG_I8K is not set
# CONFIG_X86_REBOOTFIXUPS is not set
CONFIG_MICROCODE=y
CONFIG_MICROCODE_INTEL=y
# CONFIG_MICROCODE_AMD is not set
CONFIG_MICROCODE_OLD_INTERFACE=y
CONFIG_X86_MSR=y
CONFIG_X86_CPUID=y
# CONFIG_NOHIGHMEM is not set
CONFIG_HIGHMEM4G=y
# CONFIG_HIGHMEM64G is not set
CONFIG_VMSPLIT_3G=y
# CONFIG_VMSPLIT_3G_OPT is not set
# CONFIG_VMSPLIT_2G is not set